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Color  Copy  Notice 

The  subject  matter  of  this  thesis  is  dependent  upon  the  use  of  color.  The  graphs 
utilized  in  this  study  contain  color.  The  graphs  treated  with  background  graphics  have 
their  background  area  filled-in  with  color  and  color  images.  Some  reproductions  of  this 
work  will  not  include  color.  If  the  reader  possesses  a  Defense  Technical  Information 
Center  (DTIC)  or  other  noncolor  reproduction  of  this  thesis,  a  color  copy  can  be  obtained 
by  contacting  Dr.  David  S.  Christensen  at  AFIT/LAS,  Air  Force  Institute  of  Technology, 
Building  641,  2750  P  Street,  Wright-Patterson  AFB  OH  45433-7765. 
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Abstract 

This  thesis  investigated  the  influence,  if  any,  of  background  graphics  on  the 
decision-making  process.  Specifically  the  hypotheses  tested  the  affect  of  background 
graphics  on  a  decision-makers  accuracy  and  confidence.  A  literature  review  revealed  an 
abundance  of  graphic  research  but  little  reference  to  the  use  of  background  graphics. 
Using  guidelines  previously  created  for  high-integrity  graphics,  a  timed  2x2  factorial 
experimental  design  was  developed  to  compare  the  responses  to  both  traditional  graphics 
and  those  treated  with  background  graphics.  One  hundred  forty-seven  subjects,  all 
employees  of  the  United  States  Air  Force  or  defense  contractors  were  involved  in  the 
experiments.  The  Kruskal-Wallis  test  was  employed  to  test  the  hypotheses.  The  analysis 
of  the  test  results  indicate  that  background  graphics  neither  affect  decision  accuracy  nor 
the  confidence  a  decision-maker  places  in  a  decision.  It  was  also  determined  that  gender 
and  training  have  no  affect  on  accuracy  and  confidence. 


BACKGROUND  GRAPHICS:  A  DECISION-MAKING 


HINDRANCE  OR  ENHANCEMENT  ? 


I.  Introduction 


General  Issue 

The  utilization  of  background  graphics  in  professional  presentations  has 
proliferated  over  the  last  decade.  Background  graphics  are  defined  as  any  pictures  or 
color  schemes  that  fill  in  the  background  area  behind  the  essential  information  presented  to 
a  decision-maker.  The  capability  to  place  background  graphics  on  presentations  is  now 
available  to  anyone  with  access  to  a  personal  computer  (PC).  With  the  advanced 
computer  technology  and  the  accompanying  increase  in  access  to  PCs,  anyone  with  a 
point  of  view  to  convey  can  utilize  background  graphics  to  attempt  to  influence  decision¬ 
makers.  Background  graphics  are  becoming  more  widely  used  for  the  purpose  of 
influencing  decision-makers. 

The  format  in  which  a  person  presents  the  ideas  being  advocated  is  crucial  to  the 
decision-making  process.  “Regardless  of  whom  you  are  making  a  presentation  to,  the 
purpose  of  a  presentation  isn’t  to  make  your  point,  but  to  sell  an  audience  on  your  point  of 
view!”  (Needleman,  1993:  15).  The  particular  point  of  view  in  question  is  the  presenter’s 
product.  If  the  product  is  obscured  by  the  packaging,  there  may  not  be  a  sell;  i.e.  the 
decision-maker  may  be  misled  or  distracted. 


The  outcome  of  a  presentation  depends  upon  two  main  ingredients;  the 
believability  of  the  presentation,  and  the  accurate  extraction  of  information  by  the 
decision-maker.  To  guarantee  success,  “you  not  only  have  to  understand  what 
information  you  need  to  present,  but  also  the  best  way  to  present  it  so  that  it  reinforces 
your  final  goal,  not  obscures  it”  (15).  Therefore,  the  presenter  should  choose  appropriate 
background  graphics,  if  any,  to  capture  the  decision-maker’s  attention  without  affecting 
the  outcome  (decision)  indicated  by  the  information. 

Specific  Problem 

The  marketplace  is  suffused  with  claims  about  the  power  of  graphics  to  enhance 
both  the  appeal  of  presentations  and  a  viewer’s  ability  to  use  them.  Specific  claims  have 
been  made  that  by  using  background  graphics,  “you  can  be  more  effective  in  getting  your 
point  across”  (Microsoft,  1994:  1).  Researchers  are  split  on  the  issue  of  graphics;  some 
contend  that  graphics  do  indeed  enhance  the  decision-making  process  (Horton,  1991;  12), 
while  others  reject  the  idea  (Tufte,  1983:  107  &  121).  A  third  group  of  researchers 
maintain  that  “no  empirical  evidence  exists  to  support  or  refute  this  claim”  (Jarvenpaa, 
1986:  3).  The  studies  involved  in  reaching  these  conclusions  used  experiments  that 
presented  information  to  decision-makers  in  various  forms,  but  primarily  in  graphic  or 
tabular  formats.  These  formats  are  illustrated  in  Table  1  and  Figure  1. 


Table  1.  Tabular  Presentation  of  Data 


$(000) 

1990 

1991 

1992 

1993 

1994 

1995 

In-house 
Net  Costs 

$125,117 

$122,179 

$124,938 

$128,798 

$132,112 

$136,875 

Contractor 

Bids 

$124,000 

$124,000 

$124,000 

$125,000 

$130,000 

$130,000 
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A  table  can  be  rather  short  and  narrow,  as  represented  above  in  Table  1,  or  extremely 
lengthy.  Some  tables  contain  thousands  of  entries  and  take  up  scores  of  pages.  While  not 
all  tables  are  massive  in  their  bulk,  many  are  so  large  as  to  be  incomprehensible  to  many 
potential  users.  Often  such  cases  of  excessive  tabular  bulk  can  be  reduced  to  an  easily- 
read  graphic.  Thus,  graphics  provide  a  format  for  the  utilization  of  information  previously 
underutilized,  primarily  due  to  the  cumbersome  size  of  the  information  s  format. 

Whereas  graphics  literature  is  full  of  examples  of  comparisons  between  tables  and 
graphs,  no  studies  were  found  that  examined  the  use  of  background  in  conjunction  with 
graphic  presentation  of  data.  Figure  1  below  is  an  example  of  how  graphics  can  be  used 
to  convey  informative  data. 


Total  Revenue  vs.  Total  Costs 

S(OOO)  _ _ _ 


PiRevenue  ^Costs 


1990  1991  1992  1993  1994  1995 


Figure  1.  Graphical  Presentation  of  Data 
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As  the  capability  to  put  background  behind  graphic  information  displays  has 
increased  over  the  last  several  years,  many  more  presentations  are  made  utilizing 
background  graphics.  An  example  of  background  graphics  use  is  provided  in  Figure  2.  In 
recent  years  the  access  to  background  graphics  capability  has  become  widespread.  As  an 
increasing  number  of  people  become  proficient  in  the  use  of  these  graphics  options,  some 
research  should  be  conducted  concerning  the  influence  on  decision-making  of  these 
graphics  applications. 


;  Percen  iage  .'of .  'Cafeteria ;  E.m  pi  oyees"Retaiii.ed- 

theHdspitai:if 
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Figure  2.  Background  Graphics  Application 


The  use  of  background  fill  has  been  referred  to  as  “chartjunk”  (Tufte,  1983;  107). 
Tufte  defines  chartjunk  as  “The  interior  decoration  of  graphics”  (107).  Using  Tufte’s 
definition,  one  can  view  background  graphics  as  a  form  of  chartjunk  the  purpose  of  which 


is  to  make  the  graph  look  more  professional  and  enhance  the  presentation,  not  the 
information.  “Data  graphics  should  draw  the  viewer’s  attention  to  the  sense  and  substance 
of  the  data,  not  to  something  else.”  Tufte  maintains  that  “a  large  share  of  ink  on  a  graphic 
should  present  data-information”  (91).  He  proposes  that  the  ink  used  on  graphs  should  be 
employed  in  conveying  the  information.  The  ink  that  fails  to  present  additional  data 
should  not  be  added  to  graphics.  Therefore,  he  would  not  advocate  the  use  of 
background  graphics.  However,  Tufte’s  positions  are  presented  without  empirical 
support.  The  proliferating  use  of  background  graphics,  repudiated  by  some  and  endorsed 
by  others,  suggest  a  need  for  further  research  in  this  area. 

Within  the  Department  of  Defense  (DoD),  as  is  happening  throughout  the  business 
community,  great  use  is  being  made  of  background  graphics.  Pictures  of  aircraft,  eagles, 
and  a  plethora  of  other  images  and  colors  are  being  placed  behind  graphically  displayed 
information  for  presentation  to  decision-makers.  The  motivation  for  this  increased 
utilization  seems  to  be  an  increased  perception  of  professionalism  and  subliminal  appeals 
to  the  decision-maker’s  interest. 

The  following  two  anecdotes  exemplify  the  concern  about  the  use  of  background 
graphics  in  business  presentations.  The  first  was  an  observation  made  during  a  fellow 
Lieutenant’s  unfortunate  education  in  background  graphics  application.  The  second  was 
brought  up  in  conversation  involving  this  thesis. 

Anecdote  1. 

Colonel  M  flies  a  specific  aircraft.  Major  W  works  for  Colonel  M  and 
often  presents  him  with  information  for  decision-making  purposes. 
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One  afternoon  Lieutenant  A  mentions  that  these  graphics  would  look 
better,  and  Colonel  M  might  favor  them  more,  if  the  type  of  aircraft  he 
flew  was  placed  in  the  background.  Upon  hearing  this  Major  W 
instructed  Lieutenant  A  to  find  a  picture  of  the  aircraft  Colonel  M 
piloted,  and  have  it  on  all  subsequent  graphic  slides  for  Colonel  M’s 
viewing. 

Lieutenant  A  was  on  a  treasure  hunt.  At  the  time,  the  base  had 
only  one  scanner  and  finding  a  picture  suitable  for  scanning  proved 
difficult.  Using  his  resourcefulness  Lieutenant  A  spent  approximately 
twelve  hours  locating  a  suitable  picture  and  coordinating  the  use  of  the 
base’s  sole  scanner.  These  twelve  hours  were  spent  to  obtain  for 
Colonel  M  something  he  did  not  request  and  possessed  questionable 
benefit.  Did  the  inclusion  of  an  aircraft  as  background  enhance  the 
presentation?  The  possibility  exists  that  the  aircraft’s  inclusion 
detracted  Colonel  M’s  attention  away  form  the  information  being 
presented  and  thus  hindered  his  ability  to  reach  the  appropriate 
decision. 

He  felt  that  the  use  of  background  graphics  was  an  unnecessary  enhancement  that 
added  no  value  to  the  presentation.  He  spent  extra  time  on  the  graphs,  inserting  material 
that  had  no  bearing  on  the  information  needed  by  the  decision-maker. 
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Another  case  involves  a  school  dean  who  was  being  briefed  by  a  faculty  member 
using  slides  with  background  graphics.  The  particular  background  graphic  that  was  used  is 
presented  in  Figure  3. 


«  Graduate  Instruction 
s  Faculty  Research 
f  Thesis  Research 
$  Faculty  Consultation 
e  Faculty  Development 
s?  Curriculum  Development 
i  Student  Support 


Figure  3.  Example  of  a  Criticized  Graphic. 

Anecdote  2. 

The  dean’s  comment  was  to  the  effect  that  it  appeared  to  him  that  the 
eagle  was  defecating  on  all  the  important  points  of  the  briefing.  His 
comment  makes  clear  that  his  attention  was  being  drawn  away  from  the 
information  being  presented.  Was  the  substance  of  the  briefing 
compromised?  Given  such  a  distraction,  one  may  be  concerned  that  the 
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decision-maker’s  ability  to  objectively  evaluate  the  information  was 
influenced  by  this  graphic. 

Although  the  DoD  isn’t  a  commercial  business  with  executives  in  the  traditional 
sense,  many  important  decisions  are  made  daily  that  impact  the  national  defense.  In 
commercial  businesses,  policy  makers  are  briefed  directly  by  the  field  experts.  The  sales 
director  makes  presentations  to  the  CEO.  Given  the  hierarchical  nature  of  DoD  decision¬ 
making,  the  decisions  made  by  lower  level  managers  impact  those  at  higher  levels; 
presentations  are  made  to  each  successive  layer  of  the  hierarchy.  Often  the  briefed 
become  the  briefers  at  that  next  level.  Therefore,  the  information  presented  to  DoD 
decision-makers  at  all  levels  should  be  free  of  influential  factors  that  are  not  directly 
pertinent  to  the  decision  at  hand.  If  background  graphics  affect  decision  performance,  the 
information  that  is  to  be  forwarded  may  become  adulterated  by  influential  pictures  and/or 
colors,  i.e.  background  graphics. 

The  results  obtained  from  graphics  research  can  improve  the  quality  of  decisions 
reached  throughout  the  DoD  decision-making  structures.  Research  into  the  use  of 
background  graphics  can  inform  DoD  personnel  as  to  the  effects  of  such  usage  on 
decision-makers,  allowing  them  to  better  present  information  in  ways  less  likely  to  skew 
the  proper  decisions  indicated  by  the  data.  Concurrently,  educated  decision-makers  can 
understand  the  effect  background  graphics  have  on  their  decision-making  ability,  allowing 
them  to  make  cognizant  adjustments  in  presentation  requirements. 

In  the  past,  research  has  focused  on  the  effects  of  graphical  formats  on  decision¬ 
making.  Although  claims  have  been  made  about  the  effectiveness  of  background  graphics, 
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little  research  has  been  conducted  to  determine  the  effect  of  background  graphics  on 
decision-making.  Typical  of  these  claims  is  Tufte’s  statement  that  “Chartjunk  does  not 
achieve  the  goals  of  its  propagators.  The  overwhelming  fact  of  data  graphics  is  that  they 
stand  or  fall  on  their  content,  gracefully  displayed”  (Tufte,  1983:  121).  Again, 
background  graphics  are  defined  as  any  pictures  or  color  schemes  that  fill  in  the 
background  area  behind  the  essential  information  being  presented  to  the  decision-maker. 
Thus  the  objective  of  this  thesis  is  to  determine  whether  background  graphics  negatively 
affect  a  decision-maker’s  ability  to  reach  the  appropriate  decision  indicated  by  the  data 
presented. 

Hypotheses 

To  achieve  the  objectives  of  this  study  two  hypotheses  were  identified: 

1.  Hoi:  Background  graphics  do  not  affect  decision  accuracy. 

2.  H02:  Background  graphics  do  not  affect  decision  confidence. 

The  first  hypothesis.  Hoi,  claims  that  the  use  of  background  graphics  does  not  influence  a 
decision-maker.  Figure  4  illustrates  the  contrast  between  the  use  of  background  graphics 
and  their  omission. 

The  null  hypothesis  states  that  no  difference  exists  in  a  decision-maker’s  ability  to 
reach  the  proper  conclusions  regardless  of  the  graph  presented.  The  graph  in  Panel  A  is 
identical  to  that  in  Panel  B  save  for  the  utilization  of  background  graphics  and  the 
accompanying  color  change  automatically  initiated  by  the  graphics  software. 


Panel  A 


Panel  B 


In-house  Net*  Costs  vs.  Contractor  Bids 

=  difference  between  re  venire  and  costs. 

CoSs 
S  (000) 


t990  1991  1992  1993  1994  1995 

The  cootractorhas  undetbid  our  costs  every  year. 

Disagree  Agree 


■ .  ■  ■  lii -kor.  se  Net  Cosli'  vs.  .Corakactor  Bids 

.Costs . f;  .  . .  •  ■  .  '  .■ . . 


Figure  4.  Comparison  of  Background  (jraphics  Application 


The  second  hypothesis,  H02,  asserts  that  decision-makers  are  equally  confident  of  a 
decision’s  accuracy  regardless  of  the  existence  of  background  graphics.  A  decision-maker 
faced  with  a  decision  supported  by  the  graph  in  Panel  A  would  make  the  same  decision  if 
support  were  instead  provided  by  the  graph  in  Panel  B. 

In  addition  to  the  null  hypotheses,  several  areas  for  sensitive  analyses  were 
identified.  Some  graphics  research  has  found  indications  that  gender  and  training  have  an 
influence  on  the  results  obtained.  These  two  factors  are  addressed  as  investigative 
questions  for  sensitivity  analysis  and  referred  to  as  hypotheses  three  and  four. 

3.  H03:  Gender  does  not  affect  Hoi  or  H02. 

4.  H04:  Graphics  training  does  not  affect  Hoi  or  H02. 

Analysis  will  be  done  to  determine  if  the  decision  results  differ  based  on  the  gender  of  the 
decision-makers.  Previous  research  indicates  that  gender  may  play  a  role  in  how  well 
graphical  presentation  of  information  is  interpreted  (MacKay  and  Villarreal,  1987:  544). 
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Also,  analysis  will  be  performed  to  determine  if  graphics  training  impacts  decision-making 
performance.  Training  has  been  identified  as  a  possible  influential  factor  on  the 
comprehension  of  graphical  presentations  (DeSanctis,  1984:  477).  Based  upon  the 
indications  of  previous  studies,  an  attempt  will  be  made  herein  to  establish  the  affect  of 
training  on  decision  accuracy  and  confidence  in  conjunction  with  the  use  of  background 
graphics.  Additionally,  an  attempt  will  be  made  to  determine  if  the  results  of  these 
sensitivity  analyses  are  generalizable  across  decision  tasks.  Decision  task  refers  to  the 
nature  of  the  decision  problem. 

Synopsis 

An  introduction  to  the  general  problem  associated  with  the  use  of  background 
graphics  was  provided.  The  general  case  was  narrowed  to  identify  the  specific  issue 
involved  with  the  application  of  background  graphics  to  graphic  presentations.  Several 
examples  and  anecdotes  were  given  to  illustrate  the  concerns  and  display  the  aesthetic 
differences  between  differing  presentation  formats.  Following  the  explanation  and 
examples  of  background  graphics,  the  specific  hypotheses  employed  in  this  study  were 
presented  and  discussed. 

Chapter  II,  the  literature  review,  examines  the  research  done  to  date  on  the  use  of 
graphics  in  conjunction  with  decision-making.  Several  studies  are  discussed  in  detail  as 
they  most  closely  relate  to  the  current  research  topic.  Chapter  III,  discussion  of  the 
methodology  employed,  explains  the  experimental  design,  its  validity,  applicable  concepts 
and  constructs,  as  well  as  the  construction  of  the  experimental  item.  Also  included  is  an 
explanation  an  example  of  each  graph  utilized  in  the  experiment.  Additionally,  the 


11 


administrative  procedures  utilized  in  conducting  the  experiment,  the  equipment  used,  and 
an  explanation  of  the  method  of  analysis  are  provided.  Chapter  IV,  the  findings  and 
analysis,  communicates  the  results  of  the  experiment,  utilizing  the  method  explained  in 
Chapter  III.  An  excursion  into  the  effects  of  graph  complexity  is  also  presented  followed 
by  a  concise  summary  of  the  results.  Chapter  V,  the  conclusion,  discusses  the 
experimental  results  as  they  relate  to  the  thesis’  hypotheses.  The  results  are  then 
discussed  in  the  context  of  their  impact  on  the  DoD.  Finally,  some  suggestions  are  made 
concerning  possible  topics  for  future  background  graphics  research. 


II.  Literature  Review 


A  substantial  number  of  research  efforts  have  been  conducted  concerning  the  use 
of  graphs  as  replacements  for  tables  and  narrative  text.  Far  fewer  studies  have  concerned 
the  use  of  graphs  as  decision-making  tools.  Eleven  previous  studies  were  identified  as 
dealing  primarily  with  decision-making.  The  variables  used  in  and  the  results  of  these 
eleven  are  summarized.  In  relation  to  background  graphics,  little  research  has  been 
accomplished.  The  studies  considered  to  have  primary  and  ancillary  information  pertinent 
to  the  application  of  background  graphics  were  given  a  more  thorough  discussion. 

The  generalizability  of  decision-making  studies  is  also  explored.  The  ability  to 
generalize  results  across  decision  problems  is  crucial  to  the  establishment  of  definitive 
conclusions.  The  claims  made  by  vendors  of  popular  software  packages  are  presented. 
These  vendors  promote  the  use  of  software  that  provides  numerous  background  graphic 
options. 

Graphics  Software 

Computer  graphics  software  packages  have  become  a  powerful  means  of 
communicating  in  the  modern  business  environment.  Information  is  presented  in  graphical 
form  with  elaborate  background  images  intended  to  appeal  to  decision-makers.  Software 
companies  are  less  than  subtle  in  their  implieation  that  elaborate  presentations  will 
positively  influence  the  particular  decision  at  hand  (Microsoft,  1994:  1;  Claris,  1993;  W- 
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1).  Claris  claims  that  graphics  produce  attractive  and  professional-looking  presentations. 
The  subtle  indication  is  that  without  graphics  a  presentation  is  somehow  unattractive  and 
less  professional. 

Another  claim  is  that  “you  can  create  an  even  stronger  visual  framework  by 
employing  the  two  primary  visual  elements  —  graphics  and  type.  By  balancing  text  with 
graphics,  you  give  the  reader  an  obvious  and  compelling  starting  point”  (Apple,  1991:  26). 
Microsoft,  Claris,  and  Apple  create  software  for  IBM  compatibles  and  Macintosh  systems. 
In  addition  to  the  claims,  the  background  graphics  available  in  Microsoft’s  PowerPoint 
software  provides  three  color  variations  on  fifty-five  designs.  The  overtones  of  these 
software  producers’  claims,  coupled  with  the  numerous  formats,  are  that  graphics  enhance 
the  presentation  of  information,  thus  improving  the  ability  of  decision-makers  to  perform 
their  jobs. 

The  boldest  claim  was  made  by  Decisioneering,  Inc.,  in  the  promotion  of  its 
Crystal  Ball®  software.  “Crystal  Ball  is  a  user-friendly,  graphically  oriented  forecasting 
and  risk  analysis  program  that  takes  the  uncertainty  out  of  decision-making.” 

Furthermore,  “With  Crystal  Ball,  you  will  become  a  more  confident,  efficient,  and  accurate 
decision-maker”  (Decisioneering,  1993:  1).  With  the  large  grov/th  in  the  computer 
graphics  industry,  estimated  to  be  exceeding  a  compounded  rate  of  60%  per  year 
(DeSanctis,  1984:  463;  Brown,  1984:  89;  Ward,  1992:  318),  the  effect  of  graphics  on 
decision-making  should  be  investigated.  The  extent  to  which  the  new  generation  of 
graphics  software  provides  added  value  is  a  valid  research  question  in  light  of  the  claims 
made  and  money  expended  for  this  capability. 


Previous  Research 


Many  researchers  have  investigated  and  experimented  with  the  effect  of  graphics 
on  the  decision-making  process.  A  summary  of  several  pertinent  studies  is  provided  in 
Table  2.  The  studies  in  Table  2  were  included  because  of  they  are  relevant  to  the 
advancement  of  “graphics  use”  research.  The  left  column  lists  the  names  of  researchers 
responsible  for  the  respective  studies.  The  second  and  third  columns  reflect  the 
independent  and  dependent  variables  utilized  in  each  study.  Finally,  the  right  column 
summarizes  the  results  of  the  research.  If  more  than  one  dependent  variable  was 
investigated,  each  was  numbered.  The  results  of  the  research  was  also  numbered  to 
correspond  with  the  associated  variables. 

Although  numerous  studies  have  been  conducted  concerning  the  use  of  bar  graphs 
versus  tables  for  data  comparison,  interpretation,  extraction  speed,  accuracy,  performance, 
and  preference,  significantly  less  has  been  done  in  the  area  of  graphic’s  possible  influence 
on  decision-making  as  a  whole.  While  extraction  speed,  accuracy,  and  performance  are 
measures  of  decision-making,  they  do  not  individually  possess  the  all-inclusive  essence  of 
decision-making.  The  studies  included  in  Table  2  are  representative  of,  but  not  inclusive 
of,  the  research  done  to  date  on  graphics  as  they  affect  decision-making;  each  of  these 
research  studies  utilized  one  or  more  of  the  measures  of  decision-making.  However,  none 
attempted  to  integrate  the  varying  measures  and  tasks  involved  in  decision-making  into  a 
decision  problem.  This  thesis  requires  the  use  of  presented  data  to  achieve  more  than  data 
extraction.  The  decision-maker  must  go  beyond  specific  answers  to  questions  about  each 
graph,  i.e.  extract  data.  He  must  provide  a  decision  based  on  the  information  contained  in 
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Table  2.  Summary  of  Graphics  Research 


Authors 

Independent 

Variable 

Dependent  Variable 

Results 

Tullis,  1981 

Narrative  vs.  tables 
vs.  graphs 

1.  attitude 

2.  speed 

3.  accuracy 

4.  preference 

1 .  graphics  better 

2.  graphics  better 

3.  No  effect 

4.  graphics  better 

Zmud,  etal, 
1983 

Tables  vs.  Graphs; 
Task  complexity 

Decision  quality 

Graphs  better  for  low 
complexity  tasks; 
tables  best  for  high 
complexity. 

Corbone  & 

Gorr,  1985 

Graphs  vs. 

Enhanced  Graphs 

Decision  accuracy 

No  effect 

Blocher,  et  al., 
1986 

Tables  vs.  Graphs; 
Task  complexity 

Decision  quality 

Graphs  better  for  low 
complexity  tasks; 
tables  best  for  high 
complexity. 

Dickson,  et  al., 
1986 

Tables  vs.  Graphs; 
Bar  Graphs/Line 

1.  Interpretation 
accuracy 

2.  Decision  quality 

1 .  No  efifect/No  effect 

2.  No  efifect/Line 
graphs  better 

Jarvenpaa  & 
Dickson,  1986 

Tables  vs.  Graphs 

1.  retrieval  of  information 

2.  Recall  of  information 

3.  Message  comprehension 

4.  Recognizing  trends 

5.  Recall  large  amounts  of 
data 

1 .  No  effect 

2.  No  effect 

3.  No  effect 

4.  Graphs  better 

5.  Graphs  better 

Davis,  1989 

Tables  vs.  Graphs 

Performance 

Graphs  better  only 
when  visual  cues  aid  in 
answering  questions 

DeSanctis  & 
Jarvepaa,  1989 

Tables  vs.  Graphs 

1 .  Accurate  interpretation 

2.  Incorporate  into  accurate 
judgments 

3.  Confidence 

1.  Table  better 

2.  Graph  better 

3.  No  effect 

Larkin,  1990 

High  integrity  vs. 
Misleading  Graphs 

Decision  quality 

Integrity  better 

Barber  and 

Dunn,  1992 

Iconic  vs. 

Traditional  Graphs 

1 .  Accurate  interpretation 

2.  Impression 

1 .  No  difference 

2.  Traditional  better 

Latin  and 
Villanueva, 

1994 

3D  Graphs  vs.  2D 
Graphs  vs.  Tables 

Decision  accuracy 

No  effect 
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all  the  graphs.  The  subject  becomes  a  decision  maker  determining  a  course  of  action 
rather  than  being  limited  to  choices  among  several  aspects  of  one  data  source.  True 
decision-making  involves  more  than  one  piece  of  information  from  which  to  extract  data. 

In  this  respect,  the  current  study’s  scope  differs  from  the  focus  of  previous  research. 

These  works  go  a  long  way  in  establishing  the  influence  of  graphics  in  answering 
specific  questions  concerning  a  particular  graph.  They  do  not  attempt  to  use  a  sequence 
of  graphics  to  inform  a  decision-maker,  who  must  subsequently  utilize  the  information  to 
make  a  decision  based  on  more  than  one  isolated  graphic.  Additionally,  none  of  the 
preceding  studies  addresses  the  question  of  influence  of  background  graphics  on  decision¬ 
makers. 

Latin  and  Villamieva.  In  their  1994  AFIT  thesis,  Latin  and  Villanueva  provided  a 
comprehensive  summary  of  tables  vs.  graphs  research  (Latin,  1994:  12-14).  Their 
summary  encompasses  of  all  types  of  graph  research.  The  focus  of  this  work  is  more 
narrowly  defined  around  the  possible  influence  of  background  graphics  on  decision¬ 
making.  Table  2  provides  a  good  synopsis  of  the  research  done  on  graphs  and  their  effects 
on  decision-making.  However,  no  previous  research  on  the  effect  of  background  graphics 
on  decision-making  was  discovered.  The  thrust  of  this  literature  review  will  now  turn  to 
the  decision-making  aspect  of  graphics  research  due  to  the  absence  of  specific  research 
dealing  with  background  graphics.  Because  decision-making  is  a  major  component  of  this 
study,  some  space  will  be  dedicated  to  reviewing  the  impact  of  graphics  use  in  conjunction 
with  decision  measures,  e.g.  extraction  speed,  accuracy,  and  performance. 
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Davis.  Davis  proposed  that  “the  most  efficient  form  of  presentation  is  dependent 


on  the  level  of  question  complexity”  and  “the  most  effective  form  of  presentation  is 
dependent  on  the  level  of  question  complexity”  (Davis,  1989;  498).  Complexity  is  defined 
by  Davis  as  the  amount  of  information  that  must  be  processed  by  the  decision-maker 
before  an  answer  can  be  reached.  “The  more  information  which  must  be  examined  to 
arrive  at  an  answer,  the  more  complex  is  the  question”  (497).  He  hypothesized  that  the 
more  complex  a  graph,  the  less  efficient  will  be  the  decision-maker.  Likewise,  the  more 
complex  a  graph,  the  less  accurate  will  be  decisions  reached  utilizing  the  graphic.  As 
defined  by  Davis,  complexity  is  the  amount  of  information  that  must  be  examined  to  reach 
a  conclusion.  Therefore,  a  graph  that  includes  background  graphics  (that  is,  more 
information),  might  further  denigrated  the  decision-making  task. 

In  this  study,  the  presentation  modes  (tables  vs.  graphs)  had  inconclusive  results  on 
the  questions  of  efficiency  and  effectiveness.  However,  Davis  found  an  interesting 
interaction  effect  among  the  data.  Further  analysis  seemed  to  show  that  “different  forms 
of  presentations  are  best  for  different  tasks”  (497).  Davis  indicates  that  no  absolute 
decision  concerning  the  use  of  graphs  over  tables  can  be  acknowledged.  That  is,  no  one 
particular  form  of  problem  presentation,  tabular  or  graphic,  is  best  for  all  decision 
situations.  On  the  contrary,  presentation  forms  must  take  into  consideration  the  decision 
task.  As  a  concluding  note,  Davis  points  out  that  his  findings  could  help  to  explain  the 
inconclusive  results  of  prior  research  efforts.  Davis’  research  effort  is  largely  inconclusive 
as  well,  but  does  seem  to  indicate  that  decision  accuracy  is  enhanced  when  specific. 
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decision-related  cues  are  used  in  conjunction  with  a  decision  task.  A  complete  critique  of 
Davis’  study  is  contained  in  Appendix  A 

Do  background  graphics  provide  decision  cues?  Or,  do  background  graphics 
complicate  the  question  to  be  answered  (the  task)?  The  use  of  color  schemes  and  pictures 
is  the  essence  of  background  graphics.  Do  they  influence  the  decision-making  process? 
Davis’  research  would  seem  to  indicate  that  background  graphics  may  indeed  have  an 
influence. 

The  work  of  Tufte  and  Davis  indicates  that  background  graphics  increase  the  level 
of  complexity.  The  increased  use  of  ink,  necessary  to  fill-in  backgrounds,  qualifies  as 
chartjunk  (Tufte,  1983:  107).  Background  graphics  also  contribute  to  the  quantity  of 
information  that  must  be  processed  by  a  decision-maker.  This  increase  in  a  graph’s  level 
of  complexity  reduces  the  probability  that  a  decision-maker  can  reach  an  accurate 
conclusion. 

Davis’  result  that  no  one  mode  of  presentation  is  better  in  all  decision  tasks  creates 
the  problem  of  determining  what  means  are  appropriate  for  deciding  upon  a  background 
graphic  format  in  a  given  decision  situation.  If  each  situation  is  best  supported  by  differing 
presentation  formats  with  problem  specific  cues,  how  can  a  presenter  know  which  is  most 
conducive  to  accurate  decision-making?  Additionally,  the  varying  presentation  modes 
require  that  the  decision-maker  be  proficient  in  the  interpretation  of  each. 

DeSanctis  and  Jai'venvaa.  In  their  study,  DeSanctis  and  Jarvenyaa  created  a 
conceptual  framework  to  incorporate  extraction  accuracy  and  judgments.  Judgments,  as 
constructed  for  the  purpose  of  their  study,  are  not  decisions,  but  rather  perceptual 


recognitions  of  direction  and  position  (Cleveland,  1985:  230).  Cleveland  claims  that 
accurate  judgments  about  data  “can  be  made  effortlessly  and  almost  instantaneously” 

(23 1).  DeSanctis  and  Jarvenpaa  set  out  to  test  Cleveland’s  claim.  Specifically  they  tested 
whether  “forecast  accuracy  will  be  better  when  data  is  displayed  in  a  graphical  format  than 
when  data  is  displayed  in  a  numeric  format”  (DeSanctis  and  Jarvenpaa,  1989:  511).  The 
results  of  the  study  indicate  that  forecast  accuracy  was  better  using  graphics  while  numeric 
tables  provided  more  accurate  interpretation  —  identifying  and  communicating  specific 
data  values. 

Corhone  and  Gorr.  A  natural  tendency,  given  the  positive  effect  of  graphics  on 
judgment  type  tasks,  might  be  to  improve  or  enhance  the  graphics  containing  the  data. 


Figure  5.  Enhanced  Graphic 


Improved  or  enhanced  graphs  refer  to  those  that  provide  decision  cues  or  redundant  data 
to  aid  the  decision-maker  in  arriving  at  an  appropriate  conclusion.  However,  a  study  that 
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employed  this  philosophy  concluded  that  enhanced  or  improved  graphics  did  nothing  vis- 
a-vis  conventional  graphs  to  improve  the  decision-maker’s  judgments  (Corbone  and  Gorr, 
1985:  159).  Corbone  and  Gorr’s  enhancements  were  the  mere  attaching  of  numerical 
references  to  the  graph  values  directly  on  the  graphic.  An  example  of  Corbone  and  Gorr’s 
application  of  an  enhancement  can  be  seen  in  Figure  5.  These  results  suggest  that  graph 
presentation  is  marginally  beneficial  to  a  point  beyond  which  additional  graph 
enhancements  add  no  value  to  the  task.  However,  the  enhancements  indicated  in  Corbone 
and  Gorr’s  work  do  not  include  background  graphics. 

Dickson.  De  Sanctis,  and  McBride.  More  closely  related  to  this  thesis  topic, 
Dickson,  DeSanctis,  and  McBride  conducted  an  experiment  focusing  on  “the  impact  of 
.  .  .technology  on  the  quality  of  organizational  decision-making.”  They  point  out  that 
“overall,  research  on  the  effectiveness  of  graphs  as  decision  support  tools  is  rather  sparse” 
(Dickson  et  ai,  1986:  40).  This  study  compared  the  effects  of  graphics  and  tables  on 
decision-making.  The  results  of  the  experimentation  led  to  the  conclusion  that 
“generalized  claims  of  superiority  of  graphic  presentation  are  unsupported,  at  least  for 
decision-related  activities”  (40).  The  authors  further  speculate  that  “the  experiments 
suggest  that  the  effectiveness  of  the  data  display  format  is  largely  a  function  of  the 
characteristics  of  the  task  at  hand,  and  that  impressions  gleaned  from  ‘one  shot’  studies  of 
the  effectiveness  of  the  use  of  graphs  may  be  nothing  more  than  situationally  dependent 
artifacts”  (40).  As  was  the  case  with  DeSanctis,  Jarvenpaa,  and  Corbone  and  Gorr,  the 
study  conducted  by  Dickson,  DeSanctis,  and  McBride  fails  to  address  the  effect  of 
background  graphics  on  decision-making.  The  focus  of  these  research  efforts  has  been  to 
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examine  the  effects  of  using  graphics  to  represent  data  rather  than  add  aesthetics  to 
influence  decision  efforts. 

The  idea  that  the  effectiveness  of  graphics  is  situationally  dependent  is  difficult  to 
support.  Dickson,  DeSanctis,  and  McBride  attempted  to  identify  decision  situations  or 
environments  in  which  either  tabular  or  graphical  presentation  modes  are  superior  to  the 
other.  They  concluded  that  “claims  of  total  superiority  of  graphs  should  be  viewed  with 
skepticism”  (Dickson,  et  al,  1986:  46).  These  researchers  suggest  that  the  effectiveness 
of  graphics  as  a  persuasion  tool  is  superior  to  that  of  decision  support.  This  claim  is  made 
without  the  necessary  supporting  data  and  for  the  purposes  of  suggesting  a  future  research 
topic.  Furthermore,  this  claim  is  even  more  pertinent  to  graphs  with  background  than 
those  without.  The  claims  of  background  graphics  software  producers  are  that  their 
background  graphics  features  make  graphs  more  appealing  and  professional.  The 
implication  being  that  more  professionalism,  in  the  form  of  graphics,  will  influence  the 
decision-maker. 

Graphics  Research  and  Background  Graphics 

Research  reported  by  Davis,  DeSanctis,  and  Dickson  addresses  the  use  of  graphics 
in  decision-making  tasks.  However,  they  provide  little  indication,  if  any,  as  to  the  effect  of 
using  background  graphics  when  presenting  a  decision  problem  to  a  decision-maker.  A 
primary  difference  between  the  current  study  and  the  others  summarized  above  is  that  the 
latter  merely  address  performance  in  data  extraction  task  activities.  In  these  research 
efforts,  decision  tasks  should  not  be  confused  with  decision  problems.  “Tasks,”  as  used  in 
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graphics  research,  refers  to  a  decision  between  two  or  more  aspects  of  a  graph’s  data 
(Cleveland,  1985:  231).  A  decision  problem  uses  the  information  presented  in  one  or 
more  graphs  to  facilitate  a  decision  between  alternative  courses  of  action.  In  this  respect, 
the  purpose  of  this  thesis  is  decidedly  different  from  previous  graphics  research  and 
experimentation  efforts. 

A  secondary  difference  between  this  study  and  others  previously  conducted  is  the 
essential  nature  of  a  “graphic.”  As  used  in  previous  research,  a  graphic  representation 
refers  to  data  presented  in  the  form  of  bar  charts,  pie  charts,  or  other  creative  non¬ 
narrative  forms.  The  focus  of  the  current  work  is  on  background  graphics,  defined  as  the 
use  of  images,  pictures,  and  color  schemes  to  fill  in  the  background  area  behind  the 
graphics  of  a  presentation.  The  widespread  use  of  such  background  graphics  is  relatively 
new  to  the  field,  becoming  available  only  recently  in  conjunction  with  the  proliferation  of 
inexpensive,  user-friendly  software  packages. 

On  a  tertiary  level,  the  generalizability  of  the  previous  research  has  consistently 
been  cited  as  a  definite  limitation  (Davis,  1989;  505;  DeSanctis,  1989:  523;  Dickson, 

1986:  46).  This  limitation  is  due  to  the  task  nature  of  the  experimentation.  Previous 
experiments  and  studies  have  used  several  graphic  and  table  data  sets  to  extract  answers  to 
specific  questions  concerning  particular  subject  matter.  This  work  differs  from  previous 
studies  in  its  attempt  to  generalize  across  decision  problems  in  determining  if  background 
graphics  affect  decision-making. 


Summary 


While  research  exists  recommending  both  the  use  and  non-use  of  graphics  in 
decision-making  processes,  little,  if  anything,  has  been  done  in  the  area  of  background 
graphics.  The  predominate  research  relating  to  graphics  as  an  influence  on  decision¬ 
making  was  summarized  in  Table  2.  The  studies  that  dealt  specifically  with  graphics  and 
decision-making  were  discussed  in  some  detail.  The  results  of  these  studies  were 
evaluated  in  terms  of  their  connection  to  background  graphics. 

The  bulk  of  graphics  research  has  dealt  extensively  with  the  use  of  graphs  for 
displaying  information.  Taken  as  a  whole,  the  results  of  previous  research  efforts  have 
been  inconclusive.  However,  little  attention  has  been  given  to  decision-making  tasks  as  a 
whole  versus  component  measures  of  reaching  a  decision.  Additionally,  the  use  of 
graphics  has  been  limited  to  the  graphical  display  of  information.  This  thesis  expands 
graphics  usage  to  include  all  types  of  graphical  pictures,  icons,  and  color  schemes  available 
for  incorporation  into  presentations  to  decision-makers.  This  work  is  specifically 
interested  in  determining  if  decision-making  (as  a  whole,  not  its  individual  measures)  is 
influenced  through  the  use  of  background  graphics. 


Ill,  Methodology 


The  purpose  of  this  research  effort  is  to  determine  whether  background  graphics 
affect  decision-making.  The  experimental  design  is  described.  Additionally,  a  discussion 
is  provided  concerning  the  experiment’s  construction  and  the  characteristics  of  the 
population  and  sample.  Following  this  discussion,  a  description  is  provided  of  the  support 
equipment  necessary  to  administer  the  experiment  and  analyze  the  data  obtained.  The 
latter  part  of  this  chapter  discusses  the  analysis  to  be  performed  on  the  data  to  test  the 
proposed  hypotheses. 

The  hypotheses  proposed  for  this  effort  are  these: 

Hoi:  Background  graphics  do  not  affect  decision  accuracy. 

H02:  Background  graphics  do  not  affect  decision  confidence. 

The  purpose  of  hypothesis  Hoi  is  to  test  whether  the  use  of  background  in 
conjunction  with  graphic  data  presentation  impacts  the  accuracy  of  the  decision-maker. 
For  the  purposes  of  this  study,  accuracy  is  determined  to  be  the  ability  of  the  decision¬ 
maker  to  reach  the  conclusion  indicated  by  the  data  presented  in  the  graph.  The  decision¬ 
makers  are  experimental  subjects  who  are  asked  to  draw  conclusions  based  upon  the  data 
presented  in  graphical  format. 

The  second  hypotheses,  H02,  is  to  determine  if  the  use  of  background  graphics 
influences  the  decision-maker’s  level  of  confidence  in  the  decision.  A  decision-maker’s 
confidence  level  is  self-defined;  the  subject  is  asked  to  reveal  his  or  her  confidence  in  the 


conclusion  reached.  Confidence  refers  to  the  decision-maker’s  level  of  assurance  in  the 


derived  conclusion. 

Experimental  Design 

The  design  of  the  experiment  is  2  x  2  factorial.  Factor  1  was  the  application  of 
background  graphics.  Background  graphics  was  defined  as  the  use  of  pictures  and  color 
schemes  in  the  background  of  graphically  presented  data.  The  two  levels  within  factor  1 
are  the  presence  of  background  graphics  and  no  background. 

Factor  2  was  decision  problem.  Two  separate  decision  problems  make  up  this 
factor.  The  first  problem  involved  an  outsourcing  decision,  the  second  a  benchmarking 
decision.  Other  researchers  have  referred  to  differing  problems  as  different  decision  tasks 
or  environments  (Davis,  1989  and  Dickson,  1986).  Outsourcing  refers  to  the  decision 
involved  in  determining  if  a  firm  should  continue  to  produce  a  good  or  service  internally  or 
purchase  the  product  from  an  external  source.  Benchmarking  is  the  process  of  comparing 
a  firm’s  performance,  in  a  given  area,  against  the  best  practices  among  all  firms  that 
perform  the  same  or  analogous  functions.  Typically,  benchmarking  is  done  to  determine 
whether  assistance  will  be  sought  to  improve  performance  in  the  area  being  compared. 

The  control  groups,  one  for  each  decision  problem,  were  presented  the  graphs 
without  background  graphics,  while  the  experimental  groups  were  shown  graphs 
containing  background.  Each  control  and  experimental  group  was  exposed  to  only  one 
decision  problem. 


The  design  of  this  experiment  is  illustrated  in  Figure  6.  Factor  1  is  identified  as 
“Background.”  The  lack  of  background  graphics  is  denoted  by  a  “0,”  meaning  that  no 
background  was  presented.  The  “1”  indicates  that  background  graphics  were  placed  on 
the  graphs.  Factor  two  is  classified  as  “Decision  Problem,  with  “0”  denoting  the 
outsourcing  decision  problem  and  “1”  the  benchmarking. 


Tables.  Experiment  Design 


Decision  Problem 
1  2 
Outsourcing  Benchmarking 

Background  0  No _ 

1  Yes 


The  equation  of  this  experimental  design  takes  the  following  form: 

Response  =  p  +  DP  +  BG  +  (DP*BG)  +  e 

where,  DP  =  decision  problem 

BG  =  background  graphics 

(DP*BG)  =  the  interaction  of  DP  and  BG,  and 
e  =  the  error  term. 

The  response  variables  used  to  test  Hoi  and  H02  are  decision  accuracy  and 
confidence.  Both  decision  problems  were  constructed  to  elicit  a  positive  response.  The 
decision  response  was  obtained  from  the  subjects  immediately  following  their  viewing  of 
the  graphs.  A  correct  response  was  coded  “0”  and  an  incorrect  one  “1”;  see  Appendix  G 
for  illustration.  The  subjects  indicated  either  “No”  or  “Yes”  on  the  answer  sheet  provided 
(Appendices  D  and  E). 


The  subject’s  confidence  in  their  decisions  was  ascertained  using  the  demographics 
questionnaire.  Question  10  asked: 


Based  solely  on  the  information  provided,  how  confident  are  you  that  you  made 

the  appropriate  overall,  final  decision? 

1  2  3  4  5 

6  7 

not  confident 

confident 

The  responses  of  each  of  the  subjects  were  recorded  and  used  in  the  analysis  as  their 
confidence  level.  Again,  Appendices  D  and  E  contain  the  responses  for  each  respondent. 

Experimental  Subjects 

The  population  of  this  experiment  is  the  public  in  general  because  virtually 
everyone  in  society  is  exposed  to  graphics.  Everyone  makes  decisions  based  upon  data 
presented  in  graphics  form  with  background  images  behind  the  information.  These 
decisions  range  from  purchasing  goods  using  graphically-presented  marketing  techniques 
to  learning  from  academic  materials  using  graphs  to  conserve  type-space.  This 
conservation  is  evident  when  data  from  many  pages  is  presented  in  a  single  graphic. 
Mainstream  newspapers  and  magazines  are  full  of  examples  where  people  are  presented 
with  options  augmented  by  background  graphics.  Companies  often  embed  their  logo  in 
the  background  of  their  financial  reports  as  well  as  their  advertising  campaigns. 

Air  Force  Institute  of  Technology  (AFIT)  Professional  Continuing  Education 
(PCE)  students  represented  the  general  public.  The  PCE  classes  selected  for  this  study 
were  dependent  upon  course  director  and  instructor  approval.  This  approval  was 
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dependent  upon  class  availability  and  schedule.  No  specific  classes  were  targeted  for  use 

in  the  sample.  While  a  convenience  sample  is  considered  by  some  to  be  the  least  reliable  in 

support  of  design  (Emory,  1991:  274),  it  has  been  used  in  previous  experiments  (Larkin, 

1990,  Barber  and  Dunn,  1992;  Latin  and  Villenueva,  1994).  Additionally,  the  use  of 

students  for  exploratory  “theory  building”  research  seems  appropriate. 

[F]or  activities  of  the  type  our  experiments  require,  there  is 
no  reason  to  believe  that  students  would  perform  any 
differently  than  managers.  Additionally,  for  this  kind  of 
work,  one  needs  the  power  in  tests  that  can  be  obtained  by 
the  large  numbers  available  in  student  subjects.  (Dickson  et 
al,  1986;  46) 

Therefore,  the  use  of  PCE  students  should  not  present  a  weakness  to  the  experiment’s 
design.  Moreover,  the  typical  PCE  student  is  a  management  level  decision-maker  who  is 
exposed  to  background  graphics  as  much  as  individuals  in  the  population.  By  not 
targeting  specific  PCE  courses,  the  sample  will  include  a  broader  base  of  the  population, 
i.e.  military  officers  and  enlisted  personnel,  DoD  civilians,  and  defense  contractors. 

Experiment  Construction 

Initially,  decision  problems  were  sought  that  could  be  supported  by  relevant 
background  graphics  contained  in  PowerPoint®  4.0  and  were  simple  enough  to  administer 
in  less  than  20  minutes.  This  time  constraint  was  imposed  due  to  the  nature  of  the  subject 
sample  groups.  The  PCE  classes  could  dedicate  only  a  limited  amount  of  time  to  the 
experiment.  The  decision  problems  chosen  involved  questions  concerning  outsourcing 
and  benchmarking.  These  decision  problems  were  chosen  because  they  represent  the 
decisions  currently  being  considered  in  today’s  market.  Another  reason  for  their  choice 
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was  the  existing,  accepted  measures  of  evaluating  the  criteria  involved  in  reaching  a 
decision  on  these  problems. 

Each  decision  problem  was  supported  with  six  graphs  that  led  to  an  appropriate 
decision.  The  appropriateness  of  the  decision  was  determined  based  on  the  overall 
indication  of  the  data  contained  in  each  graphic.  The  correct  decision  was  determined  as 
the  one  that  would  be  implied  solely  based  on  the  information  in  the  graphs.  The  financial 
and  performance  trends  indicate  a  clear  decision  alternative.  Additionally,  each  graph 
contained  a  statement  at  the  bottom  with  which  the  subjects  were  to  indicate  agreement  or 
disagreement.  Figures  6  through  17  indicate  the  statements  included  on  the  graphs.  The 
responses  for  each  of  the  six  graphs,  used  in  each  decision  were  consistent  across  both  the 
decision  problems;  i.e.  the  correct  response  to  each  graph  was  the  same  regardless  of  the 
decision  problem.  Both  decision  problems  consisted  of  three  agree  and  three  disagree 
graph  responses.  The  appropriate  answer  to  both  decision  problems  was  “yes.”  The 
service  provided  in  the  outsourcing  problem  should  indeed  be  outsourced  and  the  firm  in 
the  benchmarking  problem  should  benchmark  its  process.  Thus,  the  decision  problems 
were  parallel  in  construction.  The  correct  responses  for  each  graph  were  identical  across 
both  the  decision  problems,  as  well  as  the  experimental  and  control  groups. 

Explanation  of  the  graphs  used  in  the  experiment  and  the  identified  correct 
answers  are  provided  to  enhance  the  understanding  of  the  decision-making  process 
involved  with  each  decision  problem.  The  outsourcing  decision  problem  casts  the 
experiment’s  subject  as  the  Chief  Executive  Officer  (CEO)  of  a  major  metropolitan 
hospital,  whose  board  of  regents  has  charged  with  the  task  of  determining  whether  the 


hospital  should  continue  to  operate  its  cafeteria.  Based  solely  on  the  information 
contained  in  the  graphs,  the  subject  must  decide  whether  to  maintain  in-house  operation  of 
the  cafeteria  or  contract  out  its  function  to  a  commercial  food  service  contractor.  A  full 
explanation  of  the  decision  problem  may  be  obtained  from  Appendix  D. 

The  first  graph  presented  to  the  outsourcing  decision-makers  dealt  with  the 
comparison  of  in-house  operation  costs  with  the  bids  of  a  contractor  offering  to  provide 
the  service.  A  comparison  of  the  type  included  in  Figure  6  is  pertinent  to  any  decision 
concerning  the  potential  for  outsourcing.  The  contractor  has  underbid  the  in-house  costs 


In-house  Net*  Costs  vs.  Contractor  Bids 


*Net  =  difference  between  revenue  and  costs. 


Costs 
$  (000) 

140 


1990 


Disagree 


1991  1992  1993  1994  1995 

The  contractor  has  underbid  our  costs  ever^’  year. 


Hnhouse 
Net  Costs 

]  Contractor! 
Bids 


Agree 


Figure  6.  First  Outsourcing  Graph 


in  each  year  under  consideration  save  one;  therefore,  the  appropriate  response  to  this 
graph  is  “disagree.” 
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In  the  second  outsourcing  graphic,  Figure  7,  the  revenue  generated  by  the  cafeteria 
is  plotted  against  the  costs  associated  with  its  operation.  The  correct  response  would  be 
to  agree  with  the  statement  associated  with  this  information.  A  comparison  of  total 


Total  Revenue  vs.  Total  Costs 

$  (000) 


FTlRowctniia 


1990  1991  1992  1993  1994  1995 

Total  costs  have  exceeded  total  revalue  for  eadi  of  tlie  past  six  years. 
Disagree  Agree 


Figure  7.  Second  Outsourcing  Graph 


revenue  against  total  costs  is  vital  to  understanding  the  potential  for  short  run  as  well  as 
long  run  profitability.  This  understanding  serves  as  valuable  information  in  the  overall 
decision  problem. 

In  Figure  8  the  comparison  between  costs  and  revenue  is  made  in  a  different  way. 
The  differences  between  the  numerical  values  from  Figure  7  are  graphed  to  show  the 
magnitude  of  the  differences  by  year.  The  response  to  the  statement  on  the  Profit  and 
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Loss  graph  is  “agree.”  Each  of  the  first  three  graphs  speaks  to  the  financial  dimensions  of 
reaching  a  decision  concerning  outsourcing. 


Profit  and  Loss  (P&L) 

$ (000) 

-105 

-110 
-115 
-120 
-125 
-130 
-135 
-140 

Tlie  hospital  food  service  has  sustained  increased  losses  over  the  past  six  years. 
Disagree  Agree 


Figure  8.  Third  Outsourcing  Graph 


In  addition  to  the  financial  aspects  of  outsourcing,  other  factors  can  be  readily  identified. 
Figures  9  and  10  address  the  subject  of  customer  satisfaction.  Satisfaction  with  the 
internally  operated  cafeteria  has  been  on  the  decline,  but  has  not  decreased  every  year  over 
the  entire  period  under  consideration  of  this  decision.  Therefore,  the  necessary  response 
here  is  “disagree.”  Conversely,  the  potential  contractor  reports  that  customer  satisfaction, 
in  facilities  under  its  management,  is  generally  increasing.  Joined  with  the  data  in  Figure  9, 
the  non-financial  information  also  tends  to  indicate  that  the  contractor  offers  an  alternative 


Hospital  Cafeteria 
Customer  Satisfaction 

l=poor  10=  excellent 


1990  1991  1992  1993  1994  1995 


Customer  satisfaction  with  hospital  food  service  &  quality  declined  eadi  year  over  the  period  considered. 


Disagree 


Agree 


Figure  9.  Fourth  Outsourcing  Graph 


Contractor  Reported 
Customer  Satisfaction 

l=pOOr  10=  excellent 


1990  1991  1992  1993  1994  1995 

Customer  satisfaction  with  the  contractor’s  food  service  &  quality  has  consistently  declined. 
Disagree  Agree 


Figure  10.  Fifth  Outsourcing  Graph 


superior  to  continued  in-house  operation  of  the  food  service  facility.  The  logical  answer 
to  Figure  lO’s  graphic  is  “disagree.” 

Another  important  subjective,  and  perhaps  politically  motivated,  factor  in  decisions 
to  outsource  are  the  displacement  of  employees.  Therefore,  the  subject  of  the  graph  in 
Figure  1 1  is  the  continuation  of  employment.  The  contractor  has  agreed  to  “keep-on” 
90%  of  the  cafeteria  payroll,  if  allowed  to  contract  for  the  service  in  question.  The  other 
10%  are  staff  employees  who  will  be  retained  by  the  hospital.  Therefore,  no  cafeteria 
employee  is  to  lose  his  or  her  position  due  to  the  outsourcing  of  the  hospital  food  service. 
So  the  response  to  this  graphic  should  be  “agree”,  furthermore,  this  graph  supports  the 
affirmative  case  for  outsourcing. 


Percentage  of  Cafeteria  Employees  Retained 
by  the  Contractor  and  the  Hospital  if  the 
Food  Service  Function  is  Contracted  Out 

100% 
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70% 
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20% 

10% 

0% 

Contractor  Hospital  Total 

No  employees  would  lose  tlieir  jobs  as  a  result  of  contracting  out  tlie  food  service  operation. 

Disagree  Agree 


Figure  1 1 .  Sixth  Outsourcing  Graph 
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Each  of  the  six  outsourcing  graphs,  contained  in  Figures  6  through  1 1,  provides 
the  decision-maker  with  information  supporting  the  conclusion  that  outsourcing  is  the 
appropriate  decision. 

The  graphs  contained  in  Figures  12  through  17  are  those  presented  to  the 
benchmarking  decision-makers.  Each  subject  in  the  benchmarking  decision  problem  is 
cast  as  the  Chief  Executive  Officer  (CEO)  of  a  large  industrial  firm  that  produces  and 
distributes  ball  bearings.  Over  the  past  six  quarters  (1  V2  years)  it  has  come  to  the 
management’s  attention  that  something  is  wrong  with  the  firm’s  distribution  process.  No 
one  within  the  company  can  exactly  identify  the  problem.  Based  solely  on  the  information 
contained  in  the  graphs,  the  subject  must  decide  whether  to  continue  working  the  issue  in- 
house  or  consult  the  world’s  most  efficient  distributor  for  assistance  in 
evaluating/benchmarking  your  company’s  distribution  process. 

All  the  graphs  are  constructed  to  yield  the  answer  that,  yes,  the  firm  should 
benchmark  its  process.  Support  for  this  conclusion  begins  in  Figure  12  with  a 
representation  of  the  firm’s  distribution  costs.  These  costs  have  been  on  the  rise  in  recent 
quarters,  while  those  of  the  firm’s  closest  competitor  have  remained  unchanged.  Clearly, 
the  indicated  response  to  this  graphic  is  “disagree.”  The  firm’s  distribution  costs  have 
risen,  suggesting  that  some  action  is  necessary  to  control  these  costs. 

In  addition  to  the  cost  information  presented  in  Figure  12,  the  financial  strength  of 
the  firm  is  in  decline.  In  Figure  13,  the  decision-makers  are  presented  with  the  declining 
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Distribution  Cost  Per  Item 


■94  Qtr  3  ’94  Qtr  4  '95  Qtr  1  '95  Qtr  2  '95  Qtr  3  '95  Qtr  4 

Our  distribution  cost  per  item  has  remained  unchanged. 

Disagiee  Agree 


Figure  12.  First  Benchmarking  Graph 


Stock  Price  Per  Share 


Our  stock  prices  have  declined  over  the  period  mider  consideration. 
Disagree  Agree 


Figure  13,  Second  Benchmarking  Graph 
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price  of  their  stock  vis-a-vis  the  firm’s  closest  competitor.  The  stock  prices  have  been  in 
general  decline  over  the  entire  period  under  review.  The  appropriate  response  to  the 
statements  at  the  bottom  of  each  of  these  two  graphs  is  “agree.” 

An  accompanying  item  is  displayed  via  the  graphic  in  Figure  14.  The  firm’s 
market  share  has  decreased  each  of  the  last  three  quarters,  while  the  competitor’s  has  been 
increasing.  The  trends  show  a  decided  change  in  the  firm’s  relationship  with  its 
competitor. 


Market  Share 


I  Ours 


*94  Qtr  3 


•94  Qtr  4 


*95  Qtr  1 


*95  Qtr  2  *95  Qtr  3 


*95  Qtr  4 


CXir  share  of  the  market  has  been  declining  over  the  past  three  quarters. 
Disagree  Agree 


Figure  14.  Third  Benchmarking  Graph 


The  decision-making  process  is  further  swayed  in  the  direction  of  benchmarking  by 
the  information  presented  in  Figure  15.  This  figure  depicts  the  cycle  time  of  orders,  in 
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Cycle  Time 

Days  (Time  from  receipt  of  invoice  to  shipment). 


'94Qtr3  •94Qtr4  *96  Qtr  1  '95  Qtr  2  '95  Qtr  3  *95  Qtr  4 

Cycle  time  has  decreased  since  "95  Qtr  1. 

Disagree  Agree 


Figure  15,  Fourth  Benchmarking  Graph 


Inventory  Turnover  Time 


'94  Qtr  3  '94  Qtr  4  '95  Qtr  1  ’95  Qtr  2  '95  Qtr  3  '95  Qtr  4 

The  time  needed  to  go  tlirough  100%  of  stocked  inventory  is  declining. 

Disagree  Agree 


Figure  16.  Fifth  Benchmarking  Graph 
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which  there  has  been  a  drastic  increase.  The  cycle  time  has  not  improved.  In  fact,  the 
time  required  to  cycle  an  item  through  the  process  has  doubled.  Therefore,  the  correct 
answer  to  the  accompanying  statement  is  “disagree.”  Orders  previously  processed  in  one 
day  now  take  twice  as  long. 

Just  as  the  time  to  process  orders  has  increased,  so  has  the  period  required  to  turn 
inventory.  The  inventory  turnover  time  depicted  in  Figure  16  is  not  improving,  but 
deteriorating.  The  correct  response  to  this  graph  is  also  “disagree.” 

The  graphs  explained  thus  far  indicate,  through  financial  and  non-fmancial 
measures,  that  an  internal  problem  exists.  The  final  graph  (Figure  17)  conveys  the 
message  that  the  problem  has  become  noticeable  outside  the  firm.  The  firm’s  internal 


Distribution/Delivery  Time 
Customer  Satisfaction 

1=  poor  10=  excellent 
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Customer  satisfaction  with  distribution /deli  very  time  has  declined  over  the  period  being  considered. 
Disagree  Agree 


Figure  17.  Sixth  Benchmarking  Graph 


problems  are  impacting  its  customers.  The  decision-maker  should  agree  that  customer 
satisfaction  has  decreased  over  the  decision  period. 

These  trends  would  indicate  that  action  needs  to  be  taken  to  halt  the  decrease  in 
these  performance  measures  and  find  a  means  to  correct  the  perceived  problem. 
Benchmarking  is  indicated  as  most  helpful  because  no  one  within  the  organization  has 
been  able  to  identify  the  problem.  The  use  of  an  external  benchmark,  that  is  expert  in  the 
process,  seems  to  be  the  proper  decision. 

The  construction  of  each  decision  problem  and  its  accompanying  graphical 
information  was  accomplished  to  facilitate  decision-making,  thus  testing  Hypotheses  1, 
Hoi-  The  slides  were  constructed  in  PowerPoint®  in  conformance  with  the  criteria  for 
high-integrity  graphics  (Larkin,  1990:  21-22;  Christensen  and  Larkin,  1992:  131-145). 

The  graphs  themselves  were  all  bar  charts.  The  graphs  were  projected  on  a  screen  in  front 
of  the  room  using  a  LitePro®InFocus®  projection  system.  Appendix  B  contains  the  graphs 
pertaining  to  the  outsourcing  decision  problem,  while  Appendix  C  has  those  associated 
with  the  benchmarking  one.  Each  of  these  appendices  houses  the  graphs  as  seen  by  the 
studies’  subjects. 

Background  was  added  to  the  graphs  using  PowerPoint®  4.0  templates;  the 
backgrounds  used  for  the  outsourcing  and  benchmarking  decision  problems  can  be  found 
at  the  end  of  the  following,  respective  paths:  msoffice\powerpnt\template\clrovrhd\ 
medicalc.ppt  and  msoffice\powerpnt\template\clrovrhd\worldc.ppt.  Color  was  used  in  the 
experiment  to  facilitate  increased  external  validity.  Few  presentations  seen  by  this  writer 


and  this  thesis’  advisor  have  been  of  the  black  and  white  variety.  Using  color  in  the 
experiment  was  considered  crucial  to  external  validity. 

Using  the  slide  timings  option  in  PowerPoint®  tools  menu,  each  graphic  slide  was 
programmed  to  appear  for  20  seconds.  During  this  time  interval,  the  respondents  were 
asked  to  read  the  topic  of  the  slide,  determine  the  required  task,  view  the  graph,  and 
respond  using  the  answer  sheet.  The  20  second  time  limit  was  imposed  to  mimic  the  time 
constraints  present  on  decision-makers  in  the  population  environment.  The  interval 
chosen  was  to  allow  adequate  viewing  without  providing  time  for  over-analysis. 
Decision-makers  in  the  population  must  make  many  quick  decisions  without  the  benefit  of 
reflection  and  excess  thought.  The  selection  of  20  seconds  met  with  positive  feedback  by 
all  the  participants  in  the  pilot  study. 

The  Experiment 

Prior  to  the  administration  of  the  experimental  instrument,  pilot  tests  were 
conducted  on  AFIT  faculty  and  students.  These  tests  were  limited  in  that  the  number  of 
subjects  was  kept  small;  twelve  subjects  participated  in  the  pilot.  The  reason  for  this 
minimum  participation  pilot  testing  was  to  facilitate  comprehensive  feedback  from  the 
respondents.  Following  the  experiment,  each  participant  was  independently  quizzed. 

They  were  asked  questions  concerning:  (1)  the  length  of  the  decision  problem 
explanation;  (2)  the  time  constraints  placed  on  the  viewing  of  each  graph;  (3)  the  clarity 
of  the  instructions;  and  (4)  their  ability  to  execute  the  tasks  required. 


The  first  concern  was  with  the  length  of  the  decision  problem  introductions  and 
explanations.  Appendicies  D  and  E  contain  the  instruction  and  explanation  package. 
These  appendices  also  contain  the  answer  sheet  and  demographics  questionnaire  that  was 
completed  by  the  subjects.  The  length  of  the  instructions  and  explanation  of  the  decision 
tasks  and  problems  were  important.  The  success  of  the  study  was  dependent  upon  the 
attentiveness  of  the  subjects.  This  required  that  a  balance  be  reached  between  providing 
enough  information  to  facilitate  a  representative  decision  and  overloading  the  subjects 
with  so  much  explanation  that  they  became  disinterested  in  conscientious  participation. 

The  second  question  addressed  the  use  of  the  time  constraints.  The  exposure  to 
the  graphs  was  timed.  The  decision  to  present  the  information  in  this  manner  revolved 
around  eliminating  boredom  and  modeling  the  haste  with  which  decisions  are  often  made 
in  the  external  environment.  The  time  was  important  because  it  was  key  to  allowing 
sufficient  time  to  comprehend  the  information  being  presented,  while  continuing  to 
progress  the  experiment  at  a  pace  that  would  not  thwart  interest. 

In  regard  to  questions  three  and  four  as  posed  to  the  pilot  group,  it  was  necessary 
to  determine  if  the  instructions  were  clear  and  understood  by  all.  Likewise,  the  ability  to 
execute  the  tasks  asked  of  them  throughout  the  course  of  the  experiment  was  crucial  to 
the  success  of  the  study.  The  purpose  was  to  determine  if  the  experimental  instrument 
was  constructed  in  such  a  way  as  to  be  comprehensible  and  executable  by  the  intended 
subjects. 

One  major  redesign  of  the  experiment  resulted  from  the  pilot  testing.  Subjects 
tended  to  get  over-involved  in  responding  to  the  individual  graphs  and  forgot  that  a 


decision,  based  on  the  graphs,  was  required  at  the  end.  To  remedy  the  influence  of  this 
tendency  on  the  experimental  results,  a  second  section  was  added  to  the  experiment.  This 
supplementary  section  allowed  the  subjects  to  view  all  six  graphs  again  for  10  seconds 
each.  Following  this  second  viewing,  each  subject  was  again  asked  to  indicate  his  or  her 
answer  to  the  decision-making  problem. 

The  experimental  packages  were  handed  out  to  the  subjects.  The  subjects  were 
instructed  to  keep  the  packages  closed  until  told  to  begin.  Once  instructed  to  start,  the 
subjects  opened  the  packages  and  were  given  sufficient  time  to  read  the  decision  problem 
and  the  accompanying  instructions;  this  usually  took  between  three  and  five  minutes. 

Once  all  subjects  had  familiarized  themselves  with  the  problem  at  hand  and  the 
instructions,  the  experiment  was  begun.  The  graphs  were  shown  on  a  standard  wall, 
projector  screen  using  a  PowerPoint®  slide  show  running  on  an  InFocus®  LCD  projector. 
To  accompany  each  slide,  the  subject’s  packages  contained  an  answer  sheet  that  had  the 
statement  on  each  slide  reproduced  on  it.  The  subjects  viewed  the  projected  graph  and 
responded  whether  they  agreed  or  disagreed  with  the  statement  corresponding  to  each  of 
the  six  graphs.  This  process  can  be  better  understood  by  comparing  the  packages  in 
Appendices  D  and  E  with  their  respective  graphic  slides  contained  in  Appendices  B  and  C, 
respectively. 

Answer  sheets  were  handed  out  with  the  packages  so  that  each  respondent  could 
make  uninfluenced  decisions.  If  verbal  responses  were  taken,  the  bias  of  one  individual 
might  influence  others.  The  determination  was  made  to  present  the  slide  show  to  a  group 
in  a  room  versus  to  each  individual  on  a  computer  terminal  in  order  to  provide  an 


environment  more  like  the  one  in  which  decisions  of  this  nature  are  typically  made.  The 
influences  of  background  graphics  on  decision-making  are  not  as  readily  apparent  when 
sitting  inches  away  from  a  monitor  as  they  are  when  positioned  several  feet  from  the 
graphic.  After  choosing  this  method  of  answer  collection,  it  was  decided  to  provide  the 
instructions  in  the  same  format.  Therefore,  the  packages  contained  in  Appendices  D  and 
E  were  formed.  Providing  instructions  in  this  manner  reduced  the  risk  that  details  of  the 
instructions  would  be  missed  due  to  the  subject’s  proximity  to  an  information-intense 
graphic.  Thus,  the  instructions  were  provided  along  with  the  answer  sheet  and 
demographics  questionnaire  in  the  packages  rather  than  included  in  the  slide  show. 

Once  the  subjects  progressed  through  all  six  slides,  they  were  asked  to  provide  an 
answer  to  the  decision  problem.  The  next  slide  informed  the  subjects  that  they  would  be 
allowed  to  review  the  graphs  to  confirm  their  decision.  This  second  time  through  the 
slides,  the  computer  advanced  them  more  quickly  —  every  10  seconds.  The  amount  of 
time  to  review  each  slide  was  reduced  because  the  subjects  would  be  familiar  with  them 
and  no  task  was  required  --  recording  answers  —  the  second  time  the  graphs  were  viewed. 
To  this  end,  the  statements  for  agreement  or  disagreement  were  removed  from  the  graphs 
for  the  second  viewing.  At  the  conclusion  of  the  graph  review,  each  respondent  was 
asked  if  he  or  she  would  make  the  same  decision  now  as  before  the  review. 

For  the  purposes  of  categorizing  responses,  a  correct  response  in  the  review 
portion  of  the  experiment  is  considered  an  accurate  decision.  This  means  that  even  if  the 
respondent  incorrectly  answers  the  decision  problem  but  reverses  that  response  upon 
review  of  the  graphs,  the  response  will  be  considered  correct  of  the  purposes  of  this 


experiment.  The  determination  for  such  a  categorization  was  made  as  a  result  of  the 
feedback  during  the  pilot  testing.  This  adjustment  allows  for  over-involvement  in  the  data 
extraction  tasks  involved  in  the  first  viewing  of  each  graph. 

This  adjustment  provides  allowances  for  the  possibility  that  an  initial  incorrect 
decision  may  be  attributable  to  over-involvement  in  the  data  extraction  task  accompanying 
each  graph.  Such  over-involvement  would  detract  from  the  decision-making  process,  but 
not  be  attributable  to  the  experimental  treatment  being  tested.  The  second  pass  through 
the  graphs  eliminates  the  extraction  tasks  and  allows  the  subject  to  concentrate  on  the 
decision  problem.  The  influence,  if  any,  of  the  background  graphics  on  the  decision 
should  remain  throughout  both  viewings. 

Experimental  Validity 

Experimental  results  are  practically  impotent  if  they  fail  to  reflect  the  conditions  of 
the  external  environment.  Designs  that  eliminate,  to  the  maximum  extent  possible,  the 
threats  to  internal  validity  tend  to  lose  their  external  validity.  The  attainment  of  one  is 
achieved  at  some  expense  to  the  other.  Researchers  must  be  aware  of  the  tradeoff 
between  the  two  types  of  validity. 

The  existence  of  this  tradeoff  necessitates  a  discussion  of  the  threats  to  internal  and 

external  validity.  Internal  validity  questions  the  experiment’s  ability  to  accurately  estimate 

the  relationship  of  the  variables  being  tested.  The  seven  major  threats  to  internal  validity: 

1 .  History,  the  specific  events  occurring  between  the  first 
and  second  measurement  in  addition  to  the  experimental 
variable. 


2.  Maturation,  processes  within  the  respondents  operating 
as  a  function  of  the  passage  of  time  per  se  (not  specific  to 
the  particular  events),  including  growing  older,  growing 
hungrier,  growing  more  tired,  and  the  like. 

3.  Testing,  the  effects  of  taking  a  test  upon  the  scores  of  a 
second  testing. 

4.  Instrumentation,  in  which  changes  in  the  calibration  of  a 
measuring  instrument  of  changes  in  the  observers  or  scorers 
used  may  produce  changes  in  the  obtained  measurements. 

5.  Statistical  regression,  operating  where  groups  have  been 
selected  on  the  basis  of  their  extreme  scores. 

6.  Biases  resulting  in  differential  selection  of  respondents 
for  the  comparison  groups. 

7.  Experimental  mortality,  or  differential  loss  of 
respondents  from  the  comparison  groups  (Campbell  and 
Stanley,  1963:  5). 

With  respect  to  this  thesis,  history  will  not  be  a  factor  because  the  test  is 
administered  in  one  setting  with  no  allowance  for  outside  effects.  The  total  estimated  time 
of  the  experiment,  based  on  the  pilot  study,  does  not  exceed  1 5  minutes.  Therefore,  the 
effects  of  maturation  are  greatly  reduced.  No  testing  influences  are  present  since  the 
design  is  posttest  only.  No  experience  is  gained  from  a  pretest  that  might  influence  the 
experimental  or  control  test  results. 

Threats  four  and  five  are  non-factors  because  the  structure  and  content  of  the 
measurement  items  were  held  constant  throughout  the  test  packages  for  both  decision 
problems  and  between  them.  The  possible  effect  of  experimental  mortality  was  not 
expected.  Since  subjects  were  only  active  participants  in  the  experiment  for  approximately 
1 5  minutes,  the  likelihood  of  a  loss  of  respondents  is  remote. 


The  difficulties  associated  with  statistical  regression  and  selection  are  reduced 
through  randomization  (Campbell  and  Stanley,  1963:  23-24).  Subjects  were  considered 
randomly  assigned  based  on  the  random  nature  of  selection  to  attend  PCE  courses.  The 
major  commands  divide  up  the  student  slots  in  such  a  way  to  ensure/provide  random 
selection/attendance  of  PCE  courses.  Given  the  difficulty  in  administering  the  experiments 
to  ensure  external  validity  and  scheduling  agreeable  time  periods  for  test  administration, 
the  randomness  of  the  PCE  selection  process  was  assumed  adequate  for  the  experiment. 

External  validity  reflects  whether  a  relationship  can  be  generalized  across  the 
population  for  extended  periods  of  time.  Four  threats  to  external  validity  exist: 

1 .  The  reactive  or  interaction  effect  of  testing,  in  which  a 
pretest  might  increase  or  decrease  the  respondent’s 
sensitivity  or  responsiveness  to  the  experimental  variable 
and  thus  make  the  results  obtained  for  a  pretested 
population  unrepresentative  of  the  effects  of  the 
experimental  variable  for  the  unpretested  universe  from 
which  the  experimental  respondents  were  selected. 

2.  The  interaction  effects  of  selection  biases  and  the 
experimental  variable. 

3.  Reactive  effects  of  experimental  arrangements,  which 
would  preclude  generalization  about  the  effect  of  the 
experimental  variable  upon  persons  being  exposed  to  it  in 
nonexperimental  settings. 

4.  Multiple-treatment  interference,  likely  to  occur 
whenever  multiple  treatments  are  applied  to  the  same 
respondents,  because  the  effects  of  prior  treatments  are  not 
usually  erasable  (Campbell  and  Stanley,  1963:  5-6). 

A  primary  advantage  of  this  experiment’s  design  is  its  avoidance  of  reactive  or 
interactive  effects  of  testing.  Since  no  pretest  was  performed,  the  respondent’s  sensitivity 


to  the  experimental  variable  was  not  compromised.  Therefore,  the  results  of  this  design 
should  be  more  representative  of  the  population  than  any  utilizing  a  pretest-posttest 
design. 

The  effects  of  selection  biases  were  not  directly  addressed  because  of  the 
construction  of  the  experiment.  All  members  of  each  group  were  exposed  to  the  graphs  in 
the  same  order.  The  graphs  were  projected  on  a  screen  in  front  of  each  group.  So,  in 
reducing  the  reactive  effects  of  experimental  arrangements,  little  was  done  to  control 
selection  bases. 

The  reactive  effects  of  experimental  arrangements  should  be  significantly  reduced 
because  the  experiments  were  conducted  in  classroom/briefing  room  settings  that  are 
representative  of  the  non-experimental  environment.  The  treatment  was  facilitated  by 
showing  the  graphs  on  the  screen  in  front  of  the  room.  Each  subject  was  given  the 
statement  of  the  decision-problem  and  the  instructions  in  paper  form.  The  graphs  were 
shown  and  the  subject’s  recorded  responses  on  the  answer  sheet  provided.  No  graphs 
were  given  to  the  subjects  in  paper  form.  This  design  resembles,  as  closely  as  possible,  the 
nonexperimental  environment  in  existence  for  the  population  regularly  exposed  to 
background  graphics  in  decision-making  contexts. 

The  threat  of  multiple-treatment  interference  is  not  applicable  due  to  the  posttest 
only  design  of  this  experiment.  This  threat  poses  no  problem  to  the  experiments  external 
validity  as  the  subjects  were  tested  only  once. 


Other  Factors 


In  addition  to  external  and  internal  validity,  several  other  factors  should  be 
considered  in  relation  to  the  experiment.  Reproducibility  of  the  results  is  dependent  upon 
the  replicability  of  the  experiment.  To  facilitate  reproducibility  all  values  used  to  produce 
the  graphs  used  in  decision  problems  one  and  two,  i.e.  outsourcing  and  benchmarking, 
respectively,  are  contained  in  Appendix  F.  Anyone  proficient  in  the  use  of  Microsoft® 
PowerPoint®  version  4.0  and  the  data  in  the  appendix  should  be  able  to  reproduce  the 
graphs,  provided  that  they  adhere  to  the  criteria  for  high  integrity  graphics  (Larkin,  1990: 
21-22;  Christensen  and  Larkin,  1992:  131-145).  Each  graph  was  created  in  Microsoft® 
PowerPoint®  using  Larkin’s  and  Christensen’s  criteria.  If  reproduced  according  the 
aforementioned  specifications,  reproducibility  of  the  experiment  should  not  present 
difficulty. 

Reliability  of  the  experimental  item  is  unknown  because  no  other  use  of  this 
instrument  has  been  made.  Therefore,  the  consistency  of  responses  over  time,  using  this 
instrument,  is  indeterminable  at  this  time.  Additionally,  several  other  factors  impact  the 
replicability  and  reliability  of  this  experimental  instrument:  use  of  the  same  questions,  time 
constraints  on  the  subjects,  availability  of  software  employed,  and  instruction  and  response 
package.  These  additional  factors  limit  the  ability  of  other  researchers  in  attempts  at 
repeating  this  experiment.  However,  if  the  construction  and  administration  of  this 
experiment  is  duplicated  exactly  as  prescribed  in  this  text,  replicability  should  not  present  a 
problem. 
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Sensitivity  Analysis 


In  addition  to  the  two  main  hypotheses  several  other  investigative  type  questions 
will  be  addressed.  Each  of  these  questions  serve  as  sensitivity  analysis  on  the  data: 

1 )  Does  gender  affect  decision  accuracy? 

2)  Does  gender  affect  decision  confidence? 

3)  Does  graphics  training  affect  decision  accuracy? 

4)  Does  graphics  training  affect  decision  confidence? 

The  purpose  of  the  sensitivity  analysis,  investigative  questions,  is  to  determine  if  the 
experimental  results  are  sensitive  to  any  of  the  above  identified  factors.  The  first  question 
was  proposed  as  a  result  of  previous  research.  Several  studies  have  concluded  that  their 
results  are  limited  due  to  a  lack  of  generalizability  (Davis,  DeSanctis  and  Jarvenpaa, 
Dickson,  et  al.). 

This  study  attempts  to  bridge  the  generalizability  question  by  using  two  decision 
problems.  Provided  the  conclusions  from  each  problem’s  analysis  are  compatible,  some 
claim  can  be  made  that  the  results  are  indeed  generalizable.  The  remaining  six  factors 
chosen  for  the  analysis  were  selected  because  they  seemed  to  reflect  the  potential  for 
causal  impacts  on  the  hypotheses.  Questions  two  and  three  arise  from  work  done  that 
indicates  gender  differences  exist  in  the  interpretation  of  graphical  data  (MacKay  and 
Villarreal,  1987;  544). 

As  for  questions  four  and  five,  it  would  seem  plausible  that  the  more  graphics 
training  one  has  been  exposed  to,  the  better  one’s  performance  would  be  in  using  graphics 
for  decision-making  purposes.  The  level  of  graphics  training  was  ascertained  from  each 


subject  in  a  post  experiment  questionnaire  completed  by  each  subject  following  the 
administration  of  the  experiment.  The  data  collection  item  for  training  had  the  following 
form: 

Have  you  ever  had  any  training  with  graph  construction  and/or  interpretation? 

{Circle  all  that  apply). 

a)  Yes,  formal  training  on  graph  construction. 

b)  Yes,  formal  training  on  graph  interpretation. 

c)  Yes,  informal  training  on  graph  construction. 

d)  Yes,  informal  training  on  graph  interpretation. 

e)  No  formal  or  informal  training  on  graph  construction  or  interpretation. 

Each  possible  response  was  assigned  a  numerical  rating.  Response  e)  was  assigned  a 
value  of  zero,  while  responses  d),  c)  ,b),  and  a)  received  respective  values  of  1,  2,  3,  and 
4.  If  anyone  selected  more  than  one  response,  the  response  values  were  summed  to  arrive 
at  the  subject’s  training  score.  The  values  with  respect  to  responses  were  chosen  as  a 
reflection  of  the  perceived  benefit  derived  from  each  indicated  type  of  graphic  training. 
Training  should  indicate  an  increased  ability  of  a  subject  to  accurately  utilize  graphical 
data.  As  training  progresses  to  more  formal  and  involved  levels  —  construction  training 
being  more  intensive  than  interpretation  training  --  one  can  logically  postulate  that 
individuals  with  the  greatest  amount  of  training  will  perform  better.  That  is,  those  subjects 
with  a  higher  training  score  will  be  able  to  make  accurate  decisions  without  being 
distracted  by  background  images. 


Demographics  Questionnaire 


A  post  experiment  questionnaire  was  included  the  final  part  of  the  experimental 
package.  The  questionnaire  is  included  in  the  experimental  packages  in  both  Appendices 
D  and  E.  The  purpose  of  this  questionnaire  was  to  collect  demographic  data.  The 
responses  provided  by  the  subjects  on  the  questionnaire  provided  essential  information  for 
performing  analysis  on  the  experimental  results  with  respect  to  the  independent  variables. 
The  key  questionnaire  items  pertinent  to  this  thesis’  hypotheses  and  investigative  analyses 
included  questions: 

I .  What  is  your  gender? 

3.  Are  you  color  blind? 

4.  Have  you  ever  had  any  training  with  graph  construction  and/or 

interpretation? 

10.  Based  solely  on  the  information  provided,  how  confident  one  you  that 

you  made  the  appropriate  overall,  final  decision?  and 

I I .  Did  you  have  any  previous  knowledge  of  this  experiment? 

Question  one  is  pertinent  to  performing  gender  analysis  as  discussed  in  investigative 
question  one.  Determining  if  any  subjects  were  color  blind  seemed  prudent  since  color 
was  used  throughout  the  experiment.  If  any  subjects  indicate  color  blindness,  their 
responses  will  be  checked  to  determine  if  the  visual  challenge  hampered  their  ability  to 
perform  the  tasks  and  arrive  at  the  proper  decision. 

Question  four  was  used  to  collect  the  necessary  data  to  perform  sensitivity  analysis 
based  upon  training.  This  analysis  was  previously  explained  above.  The  method  of 
compiling  responses  to  this  question  was  also  explained  previously.  Question  eleven  was 


included  to  facilitate  the  exclusion  of  any  subjects  possessing  prior  knowledge  of  the 
experiment’s  purpose. 

Finally,  question  ten  was  the  avenue  of  collecting  the  data  necessary  to  test 
hypotheses  2,  H02.  The  response  options  took  the  form  of  a  7-point  Likert  scale.  The 
responses  of  each  subject  were  recorded.  This  raw  number  was  used  in  the  analysis  to 
determine  if  a  significant  difference  exists  between  the  control  and  experimental  groups. 
Ho2  is  designed  to  test  for  a  difference  between  the  confidence  level  of  subjects  whose 
decision  problem  contained  background  graphics  and  those  that  were  void  of  background 
graphics. 

Several  other  questions  were  included  on  the  questionnaire  to  provide  for  possible, 
future  interests  in  other  areas  of  sensitivity  analysis  that  might  arise  following  the 
completion  of  the  experimental  treatments  and  manipulations.  These  additional  questions 
may  also  aid  other  researchers  interested  in  the  effects  of  background  graphics  on 
decision-making. 

Equipment 

The  administration  of  the  experiment  required  the  use  of  some  specific  equipment. 
This  equipment,  including  software,  was  chosen  to  reflect  the  external  environment.  The 
equipment  utilized  in  the  experiment  is  similar  to  that  in  use  by  business  and  DoD 
decision-makers. 

The  graphs  were  constructed  using  PowerPoint®  4.0  software  running  on  a 
Hewlett-Packard  Pentium  60  micro-processor.  The  graphs  were  projected  on  the 


overhead  projector  screen  using  an  InFocus®  Systems  LitePro®  560/570  LCD  Projector. 
This  projector  was  interfaced  with  a  486  laptop  computer.  The  interface  was  set  to  run 
the  PowerPoint®  slide  show  on  both  the  laptop  and  the  projector,  which  simultaneously 
projected  the  graphs  onto  the  screen.  The  PowerPoint  file  containing  the  graphs  was 
copied  onto  the  laptop’s  c; drive.  The  file  was  opened  and  the  slide  show  was  run, 
utilizing  the  built-in  timings.  Microsoft®  Word  was  used  to  create  the  package  passed  out 
to  the  subjects.  Statgraphics  was  used  to  perform  the  analysis  of  the  collected  data.  A 
color  printer  is  necessary  to  produce  the  graphs  on  paper.  However,  paper  copies  of  the 
graphs  were  not  provided  to  the  subjects  in  the  experiment.  The  printing  of  the  graphs 
was  strictly  for  their  inclusion  in  this  thesis  text. 


Method  of  Analysis 

Analysis  of  the  obtained  data  was  accomplished  using  the  Kruskal-Wallis  test.  The 
use  of  a  non-parametric  test  vis-a-vis  a  parametric  one  was  due  to  the  nature  of  the  two 
tests. 


Parametric  tests  are  the  more  powerful  because  their  data 
are  derived  from  interval  and  ratio  measurements. 
Nonparametric  tests  are  used  to  test  hypotheses  with 
nominal  and  ordinal  data.  Parametric  techniques  are  the 
tests  of  choice  if  their  assumptions  are  met.  (Emory,  1991: 
529) 

The  aforementioned  assumptions  are  five  in  number; 

1 .  The  observations  must  be  independent. 

2.  The  observations  should  be  drawn  from  normally  distributed 
populations. 
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3.  These  populations  should  have  equal  variances. 

4.  The  measurement  scales  should  be  at  least  interval  so  that  arithmetic 
operations  can  be  used  with  them. 

5.  The  means  of  these  normal  and  homoscedastive  populations  must  be 
linear  combinations  of  effects  due  to  columns  and/or  row.  (Emory, 
1991;  529-530) 

These  assumptions  must  be  met  for  any  parametric  test  results  to  be  valid;  that  is, 
conclusive. 

In  cases  where  not  all  of  the  above  assumptions  are  met,  nonparametric  tests 
provide  an  alternative  for  evaluating  data.  For  the  purposes  of  this  thesis’  analysis,  the 
Kruskal-Wallis  nonparametric  test  was  used.  A  nonparametric  test  was  chosen  for  the 
following  three  reasons. 

1 .  Non-continuous  data.  An  overwhelming  majority  of  the  response 
variables  were  categorized  as  either  Os  or  Is.  Thus  the  data  are  of  an 
ordinal  and  nominal  nature. 

2.  The  existence  of  normality  is  questionable. 

3.  The  existence  of  equal  variances  across  groups  is  likewise 
questionable. 

The  decision  to  use  a  nonparametric  test  in  general  and  the  Kruskal-Wallis  test  in 
particular  was  based  upon  the  nominal  and  ordinal  nature  of  the  collected  data.  The 
Kruskal-Wallis  test  as  contained  in  and  employed  by  the  software  package  Statgraphics® 
version  7  (for  DOS)  was  utilized.  The  equations  and  methodology  used  in  the  Kruskal- 
Wallis  test  are  explained  in  the  Statgraphics®  user’s  manual  and  elsewhere  (Statgraphics, 
1993:  K-19  -  K-23  and  Conover,  1980:  229-239). 


To  perform  a  Kruskal-Wallis  test  within  Statgraphics®,  one  selects  the  analysis  of 
variance  option  under  the  models  menu.  Then  select  the  Kruskal-Wallis  Test.  Upon 
making  this  selection,  the  input  screen  appears.  Here  the  response  variable  data  and  the 
sample  from  which  each  observation  was  derived  are  entered  for  analysis  (Statgraphics, 
1993:  K-21).  Table  4  contains  the  summary  of  categories  of  data  that  were  entered  into 
the  software  package  for  performance  of  the  statistical  analysis.  The  data  corresponding 
to  the  Table  4  categories  are  provided  in  Chapter  IV. 

The  numerical  and  categorical  data  that  were  entered  into  Statgraphics®  is 
provided  in  Appendix  G,  while  the  actual  files  imported  into  Statgraphics®  is  contained  in 
Appendices  H  and  I.  The  format  and  procedures  for  importing  data  into  Statgraphies®  is 
explained  in  the  Statgraphics  reference  manual  (Statgraphics,  1993).  The  Statgraphics 
input  in  Appendices  H  and  I  are  a  consolidation,  by  decision  problem,  of  the  control  and 
experimental  group  data  contained  in  Appendix  G. 

The  terms  in  Table  4  are  consistent  with  their  use  throughout  this  thesis. 
Outsourcing  and  benchmarking  refer  to  the  respective  decision  problems  utilized  in  the 
experiments.  Confidence  and  Decision  represent  the  response  in  relation  to  hypotheses 
Hoi  and  H02,  respectively.  The  response  “Decision”  was  collected  after  the  initial  viewing 
of  the  graphs.  “Reviewed  Decision”  refers  to  the  response  collected  following  the,  one 
time,  review  of  the  graphs. 

Ho3  and  H04  are  the  respective  tests  regarding  gender  and  training  responses. 

These  responses  were  gathered  using  questions  1  and  4  of  the  demographics  questionnaire 
and  are  represented  as  Q1  and  Q4.  The  term  “Group,”  in  the  table  represents  the 


Table  4.  Summary  of  Data  Categories 


Decision  Problem 

Response  Variable 

Sample 

Outsourcing 

Confidence 

Group 

Outsourcing 

Decision 

Group 

Outsourcing 

Confidence 

Q1  (Gender) 

Outsourcing 

Decision 

Q1  (Gender) 

Outsourcing 

Confidence 

Q4  (Training) 

Outsourcing 

Decision 

Q4  (Training) 

Outsourcing 

SI 

Group 

Outsourcing 

S2 

Group 

Outsourcing 

S3 

Group 

Outsourcing 

S4 

Group 

Outsourcing 

S5 

Group 

Outsourcing 

S6 

Outsourcing 

Reviewed  Decision 

Group 

Outsourcing 

Reviewed  Decision 

Q1  (Gender) 

Outsourcing 

Reviewed  Decision 

Q4  (Training) 

Benchmarking 

Confidence 

Group 

Benchmarking 

Decision 

Group 

Benchmarking 

Confidence 

Decision 

Benchmarking 

Confidence 

Q4  (Training) 

Benchmarking 

Decision 

Q4  (Training) 

SI 

Group 

S2 

Group 

Benchmarking 

S3 

Group 

Benchmarking 

S4 

Group 

Benchmarking 

S5 

Group 

Benchmarking 

S6 

Group 

Reviewed  Decision 

Group 

Benchmarking 

Reviewed  Decision 

Q1  (Gender) 

Benchmarking 

Reviewed  Decision 

Q4  (Training) 

application  of  background  graphics.  “Group”  includes  the  experimental  and  control 
subjects  responses.  The  terms  SI  through  S6  denote  the  responses  to  the  statements 


graphs  one  through  six.  The  terms  are  used  in  identical  fashion  for  each  decision  problem. 
The  response  values  for  each  term  are  provided  in  Appendices  G,  H,  and  I. 

Summary 

The  methodology  used  in  the  design  and  construction  of  this  experimental  study 
was  provided.  This  methodology  incorporated  a  discussion  of  the  population,  the  sample, 
and  the  equipment  used  to  conduct  the  experiment.  Discussion  was  also  given  to  the 
tradeoffs  between  external  and  internal  validity. 

The  use  of  the  posttest  demographic  questionnaire  to  collect  responses  for  analysis 
was  explained.  Also  explained  was  the  method  used  for  applying  sensitivity  analysis  to  the 
proposed  hypotheses.  In  addition  to  the  discussion  on  sensitivity  analysis,  the  method  and 
procedure  for  analyzing  the  response  data  was  justified.  The  Kruskal-Wallis  test  yielded 
the  results  analyzed  in  the  following  chapter. 


IV.  Analysis  and  Findings 


Two  decision  problems  were  used  in  this  thesis.  Each  was  independently  used  to 
test  the  effect,  if  any,  of  background  graphics  on  decision-making.  The  data  obtained  was 
used  to  perform  analysis  with  respect  to  the  proposed  hypotheses. 

The  results  of  the  analyses  are  summarized  in  the  tables  below.  The  subject’s 
responses  are  recorded  in  Appendix  G.  Appendix  J  contains  the  statistical  output 
produced  by  StatgraphicS(8. 

Detailed  analysis  of  each  hypothesis  is  provided.  In  addition  to  the  analysis  and 
presentation  of  this  work’s  findings  with  respect  to  its  formal  hypotheses,  an  excursion 
into  the  influence  of  graph  complexity  was  undertaken.  Finally,  the  results  are 
summarized  into  three  levels  of  analysis  separating  the  concepts  of  graph  complexity, 
sensitivity  analysis,  and  the  two  hypotheses  involved. 

Hypotheses 

The  two  hypotheses  identified  to  achieve  the  objectives  of  this  study  were: 

1.  Hoi:  Background  graphics  do  not  affect  decision  accuracy. 

2.  H02:  Background  graphics  do  not  affect  decision  confidence. 

The  first  hypothesis.  Hoi,  specifies  that  the  use  of  background  graphics  do  not  influence 
decision-makers.  The  second  hypothesis,  H02,  asserts  that  decision-makers  are  equally 
confident  of  a  decision’s  accuracy  regardless  of  the  existence  of  background  graphics. 


In  addition  to  the  null  hypotheses,  several  areas  for  sensitive  analyses  were 
identified.  These  areas  were  treated  as  investigative  questions  and  referred  to  as 
hypotheses  three  and  four. 

3.  H03:  Gender  does  not  affect  Hoi  or  H02 

4.  H04;  Graphics  training  does  not  affect  Hoi  or  H02. 

Additionally,  attention  was  given  in  the  study’s  design  to  make  generalization  across 
decision  problems  a  viable  activity. 


Experimental  Results 

The  results  of  the  statistical  analyses  are  provided  in  Tables  5  and  6  along  with  the 
raw  data  associated  with  each  response  variable.  The  raw  numbers  indicate  the  number  of 
correct  responses,  save  in  the  case  of  confidence.  The  confidence  number  is  the  sum  of 
the  subject’s  responses  to  question  ten  on  the  demographics  questionnaire. 

Table  5  contains  the  raw  data  collected  from  the  outsourcing  decision  problem 
accompanied  by  the  associated  statistical  results. 


Table  5.  Summary  of  Outsourcing  Raw  Data  with  Results 


Response 

Variable 

No 

Background 

Percent 

Correct 

Background 

Percent 

Correct 

p-value 

Decision 

24 

82.76% 

23 

74.19% 

.312401 

Reviewed  Decision 

20 

68.97% 

26 

83.87% 

.761708 

Confidence 

5.21 

- 

5.20 

- 

.857429 

The  analysis  for  each  decision  problem  will  be  done  separately  beginning  with  that  for  the 
outsourcing  decision  problem.  With  respect  to  Hoi,  the  resulting  p-value  was  .3 12401; 
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therefore  Hoi  is  not  rejected.  Likewise,  H02  is  not  rejected  based  upon  a  p-value  of 
.857429.  The  inability  to  reject  either  of  the  null  hypotheses  supports  the  claim  that 
background  graphics  do  not  affect  decision  accuracy  or  confidence  in  a  decision. 

The  results  were  not  sensitive  to  gender  and  training.  The  p-values  for  subject 
confidence  as  impacted  by  their  gender  and  training  are  lower  than  those  for  decision 
accuracy,  but  still  insignificant.  Although  claims  exist  that  gender  can  make  a  difference, 
this  finding  is  not  at  all  surprising.  The  results  proclaiming  that  graphics  training  has  no 
effect  on  decision  accuracy  or  confidence  are  somewhat  surprising,  especially  at  the  level 
indicated  by  this  analysis. 

The  analysis  of  the  benchmarking  problem’s  data  yielded  analogous  conclusions 
with  respect  to  the  hypotheses.  The  raw  data  is  provided  in  Table  6.  A  p-value  of 
.563561  precluded  the  rejection  of  Hoi.  H02  was  not  rejected  based  on  a  .409316  p-value. 
The  inability  to  reject  either  of  the  null  hypotheses,  based  upon  their  insignificant  p-values, 
supports  the  claim  that  background  graphics  do  not  affect  decision  accuracy  or  confidence 
in  a  decision. 


Table  6.  Summary  of  Benchmarking  Raw  Data  with  Results 


Response 

Variable 

No 

Background 

Percent 

Correct 

Background 

Percent 

Correct 

p-value 

Decision 

36 

87.81% 

42 

91.30% 

.563561 

Reviewed  Decision 

37 

90.24% 

39 

84.78% 

.85641 

Confidence 

5.35 

- 

5.20 

- 

.409316 

With  respect  to  the  gender  and  training  sensitivity  testing,  each  was  shown  to  have 
no  effect  on  confidence  or  decision  accuracy.  Decision  accuracy  as  a  response  to  gender 


yielded  a  p-value  of  .959865;  a  p-value  of  .186988  was  obtained  in  relation  to  decision 
confidence.  Decision  accuracy  tested  against  training  with  a  p-value  of  .829885.  The 
confidence  response  approached  an  acceptable  level,  but  at  .0888819  is  outside  a  95% 
confidence  level. 

Given  that  the  initial  decisions  did  not  lead  to  the  rejection  of  any  of  the  proposed 
hypotheses,  allowing  for  decision-makers  to  change  incorrect  decisions  to  correct  ones 
could  not  change  the  conclusions  of  the  analysis.  In  fact,  the  results  of  these  tests  show  an 
increase  in  the  p-values.  This  increase  was  expected  since  those  subjects  with  incorrect 
decisions  would  use  the  second  opportunity  of  viewing  the  graphs  to  rectify  their  mistake. 

Based  on  the  comments  garnered  in  the  pilot  study,  the  possibility  exists  that  some 
decision-makers  (subjects)  became  so  absorbed  in  the  data  extraction  tasks  on  each  graph 
that  they  forgot  the  true  task  at  hand.  Becoming  so  immersed  in  the  individual  graphs, 
they  might  not  have  interpreted  the  information  in  terms  of  the  decision  problem.  The 
review  portion  of  the  experiment  was  meant  to  compensate  for  this  potentiality.  The  real 
intent  was  to  provide  explanatory  ability  in  the  event  that  the  initial  decision  tested 
significant  and  the  review  insignificant. 

In  comparing  the  p-values  of  each  of  the  benchmarking  problem’s  hypotheses  and 
sensitivity  cases,  all  save  one  showed  an  increase  from  the  initial  decision  to  the  review. 
The  p-value  for  gender  decreased  from  .959865  to  .808978.  This  decrease  between  males 
and  females  was  attributable  to  twice  as  many  of  the  former  changing  to  the  incorrect 
response.  This  happenstance  widened  the  difference  in  responses  between  males  and 
females. 
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A  similar  widening  of  differences  also  occurred  between  the  initial  and  reviewed 
decisions  on  the  outsourcing  decision  problem.  The  initial  p-value  dropped  from  .630425 
to  one  of  .3 1 5256  for  the  reviewed  decision.  However,  in  the  outsourcing  problem  more 
males  changed  to  the  correct  decision  while  four  times  as  many  females  reversed  their 
correct  decision  upon  review. 

Taken  together,  the  results  of  the  two  parallel  experimental  designs  indicate  that 
background  graphics  have  no  effect  on  decision-making  ability.  The  subjects  in  the  study 
were  equally  able  to  process  the  information  contained  in  the  graphs  irrespective  of  the 
existence  of  background  graphics.  The  presence  of  pictures  and  color  schemes  did  not 
hinder  the  decision-maker’s  ability  to  rationally  decide  upon  a  course  of  action,  i.e.  make 
the  appropriate  decision. 

The  results  of  the  analysis  are  consistent  for  both  decision  problems,  providing 
some  support  for  the  proposal  that  the  results  of  the  analysis  are  generalizable  across 
decision  problems.  However,  for  this  conclusion  to  be  definitive,  further  parallel 
experiments  should  be  conducted.  The  preliminary  indications  would  imply  that  it  is 
possible  to  generalize  across  decision-making  problems  when  addressing  subjects  dealing 
with  the  use  of  background  graphics. 

Graph  Complexity 

The  issue  of  complexity,  in  relation  to  graphs,  was  introduced  in  Chapter  II. 
Davis’  research  into  issues  of  complexity  was  discussed  previously  and  a  critique  of  his 
article  dealing  with  the  subject  is  provided  in  Appendix  A.  Although  this  study  was  not 


initially  designed  with  complexity  in  mind,  the  attempt  was  made  to  analyze  the  data  in  this 
vein.  The  response  data  obtained  (Appendix  G)  for  each  graph  statement,  whether  the 
subject  agreed  or  disagreed,  was  analyzed.  The  raw  data,  indicating  the  number  of 
correct  responses,  for  the  statement  on  each  graph  are  contained  in  Tables  7  and  8.  Also 
included  in  the  tables  are  the  statistical  results  of  the  analysis  comparing  graph  responses 
to  background  graphics  application.  These  p-values  represent  the  following  complexity 
hypothesis: 

He:  Background  graphics  do  not  affect  complexity. 

This  hypothesis  should  be  interpreted  to  mean  that  background  graphics  do  not  compound 
the  complexity  of  a  graph. 

The  rationale  for  an  analysis  of  complexity  within  this  study’s  design  exists  in  the 
fact  that  each  decision  problem’s  graphical  presentations  had  a  mixture  of  graphs,  some  of 
which  were  more  complex  that  others.  In  the  outsourcing  decision  problem,  graphs  one 
and  two  are  more  complex  than  three  through  six.  See  Appendices  B  and  F  for  visual 
verification  and  graph  titles,  respectively.  The  response  to  each  of  the  six  graphs’ 
statements  is  represented  in  Table  7  as  SI  through  S6.  The  p-values  for  SI  and  S2 
indicate  that  one  is  not  significant  and  two  has  some  significance.  However,  for  He  to  be 


Table  7.  Complexity  and  Outsourcing  Graphs 


Response  Variable 

No  Background 

Background 

p-value 

SI 

21 

19 

.273322 

S2 

29 

27 

.047179 

S3 

29 

28 

.0726449 

S4 

21 

18 

.175659 

S5 

28 

28 

.304231 

S6 

15 

12 

.137885 
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rejected,  both  should  test  significant.  Additionally,  S3  tests  significant  if  the  95% 
confidence  level  being  used  here  is  relaxed  slightly,  while  the  companion  non-complex 
graphs  show  no  significance. 

For  the  benchmarking  decision  problem,  graphs  1,  2,  and  3  are  complex.  SI 
through  S6  in  Table  7  represent  the  responses  to  each  of  the  six  graphs’  statements. 


Table  8.  Complexity  and  Benchmarking  Graphs 


Response  Variable 

No  Background 

Background 

p-value 

SI 

35 

38 

.728335 

S2 

36 

40 

.905936 

S3 

25 

39 

.0124535 

S4 

38 

38 

.17323 

S5 

36 

42 

.594775 

S6 

36 

46 

.0152843 

The  analysis  has  analogous  results  to  that  immediately  above  in  the  outsourcing  problem. 
Two  of  the  three  complex  graphs’  responses  show  no  effect  while  S3  indicates  an  effect. 
Again  mirroring  the  above  results,  the  non-complex  graphs  have  one  response,  S6,  that 
tests  significant.  These  results  do  not  coincide  with  the  hypothesis.  The  more  complex 
graphs  should  show  a  significant  difference  between  those  with  background  graphics  and 
those  without  them.  The  significant  result  in  S6  compounds  the  issue,  as  it  is  not 
considered  one  of  the  complex  graphs. 

Given  the  inconsistent  and  thus  inconclusive  p-values.  He  cannot  be  rejected. 
Therefore,  no  conclusion  can  be  reached  concerning  the  relationship  between  background 
graphics  and  complexity. 
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Summary  of  Analysis 


The  primary  analysis  was  done  on  the  two  hypotheses  that  background  graphics  do 
not  aflfect  decision  confidence  or  accuracy.  The  results  of  this  analysis  precluded  the 
rejection  of  the  null  hypotheses.  This  conclusion  was  reached  for  both  decision  problems. 

On  a  secondary  leyel,  sensitiyity  analysis  was  done  to  determine  if  gender  or 
training  impacted  the  two  primary  hypotheses.  The  analysis  suggested  that  neither  gender 
nor  training  influences  decision  accuracy.  Again,  this  conclusion  was  reached  for  both 
decision  problems. 

A  tertiary  analysis  was  accomplished  using  the  concept  of  complexity.  The  results 
of  this  analysis  proyed  to  be  inconclusiye. 
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V.  Conclusion 


Background  graphics  are  a  readily  available  means  of  adding  color  and  pictures  to 
graphic,  as  well  as  nongraphic,  presentations.  Opinions  differ  as  to  whether  background 
graphics  serve  an  enhancing  role,  a  hindering  one,  or  no  role  at  all  in  the  decision-making 
process.  The  addition  of  color  schemes  and  pictures  add  more  ink  to  the  graphs, 
surrounding  the  informative  data  with  eye  catching  images.  Some  people  prefer  the  use  of 
these  aesthetics  while  others  take  offense.  The  two  anecdotes  included  in  the  introduction 
to  this  thesis  serve  to  illustrate  the  opposing  points  of  view. 

Can  the  use  of  background  graphics  influence  the  decision-making  process?  This 
possibility  was  the  central  theme  of  this  study.  Background  graphics  has  this  potentiality. 
The  decision  produced  by  a  presentation  is  dependent  upon  two  main  factors:  the 
believability  —  the  confidence  in  the  information  provided  —  and  the  accurate 
interpretation  of  the  data.  Therefore,  the  preparation  of  graphical  data  should  attempt  to 
convey  the  required  information  without  distracting  the  decision-maker. 

Summary  of  Results 

A  review  of  the  graphics  literature  revealed  few  studies  that  focused  on  graphics 
versus  decision-making  and  no  specific  research  into  the  effects  of  background  graphics. 
The  bulk  of  previous  graphics  research  dealt  extensively  with  the  use  of  graphs  versus 
tables  for  displaying  information.  The  cumulative  results  of  previous  research  efforts  have 


been  inconclusive.  The  use  of  graphics  has  been  limited  to  the  graphical  display  of 
information.  This  study  expanded  the  use  of  graphics  to  included  all  types  of  graphical 
pictures,  icons,  and  color  schemes  available  for  incorporation  into  presentations. 

One  hundred  forty- seven  PCE  students  participated  in  the  experiments  to 
determine  if  decision-makers  are  affected  by  background  graphics.  Two  decision 
problems  were  used  independently  of  each  other  to  test  for  effects  on  decision-making 
caused  by  background  graphics.  Tables  5  and  6  contain  the  results  of  the  analyses.  The 
results  of  the  study  show  no  effect  caused  by  background  graphics. 

Impact  on  the  DoD 

Given  the  hierarchical  nature  of  DoD  decision-making,  the  decisions  made  by 
lower  level  managers  impact  those  at  higher  levels;  presentations  are  made  to  each 
successive  layer  of  the  hierarchy.  Many  of  these  presentations  contain  some  form  of 
background  graphic.  Often  the  briefed  become  the  briefers  at  the  next  level,  thus 
delivering  the  presentation  format  and  graphics  to  increasingly  important  decision-makers. 
Therefore,  the  information  presented  to  decision-makers  at  all  levels  should  be  free  of 
influential  factors  that  are  not  directly  pertinent  to  the  decision  at  hand. 

The  results  of  this  study  indicate  that  the  use  of  background  graphics  does  not 
hinder  the  ability  of  decision-makers  to  reach  accurate  decisions  and  reach  them  with 
confidence.  These  results  are  pertinent  to  decision-makers  at  all  levels  of  the  DoD 
decision-making  hierarchy.  The  results  indicate  that  presentations  using  background 
graphics  do  not  affect  the  outcome  of  the  decision-making  process.  However,  the  results 


contained  herein  are  from  only  two  experiments,  possessing  identical  design.  While  the 
results  suggest  generalizability  across  decision  problems  or  environments  is  possible,  more 
conclusive  research  needs  to  be  done  to  confirm  these  results. 

Further  Research  Recommendations 

Several  areas  for  further  research  became  apparent  during  the  course  of  this  study 
of  graphics.  While  this  study  continued  to  utilize  some  data  extraction  in  conjunction  with 
the  attempt  to  test  deeision-making,  future  research  should  sever  the  link  established 
herein.  One  suggestion  is  attempting  to  mirror  this  work’s  goal  without  the  statements  on 
each  graph.  The  only  response  required  from  the  subjects  would  be  the  decision.  This 
decision  could  come  after  extensive  viewing  of  graphs  related  to  some  decision  problem. 
Eliminating  the  data  extraction  task  associated  with  each  graph  would  exclude  the 
respondent’s  need  to  focus  on  each  graph.  This  elimination  of  the  extraction  task  might 
more  closely  emulate  the  external  decision-making  environment. 

Another  area  worthy  of  study  is  the  deeision  style  of  the  decision-maker.  To 
accomplish  this  study,  a  personality  type  indicator  could  be  given  prior  to  an  experiment 
and  subsequently  compared  to  the  response  to  see  if  decision  style  influences  decision¬ 
making.  A  focus  of  this  nature  would  place  the  study  in  the  general  area  of  studies  done 
on  risk  behavior.  The  risk  level  indicated  by  a  pre-administered  test  could  be  coupled  with 
decision  options  involving  graduated  levels  of  risk.  Thus  decision  style  and  risk  avoidance 
could  be  directly  related  to  the  decision-making  process. 


Additional  research  could  also  be  done  on  the  subject  of  complexity.  Even  though 
the  preliminary  results  of  this  thesis’  analysis  was  inconclusive,  a  proper  experimental 
design  incorporating  questions  of  complexity  in  its  hypotheses  would  stand  a  better  chance 
of  significant  findings.  The  analysis  herein  accomplished  with  regard  to  complexity  was 
done  as  an  interesting  factor,  not  as  part  of  the  deliberate  design.  Designing  an 
experiment  around  the  issue  of  complexity  would  be  a  worthy  pursuit. 


Appendix  A.  Article  Critique 

A  critique  of  Larry  R.  Davis’  article  “Report  Format  and  the  Decision  Maker’s  Task:  An 
Experimental  Investigation,”  Organizations  and  Society,  Volume  14  Number 

5/6  1989:  495-508. 


Executive  Summary 

This  critique  summarizes  and  evaluates  Davis’  experimental  investigation  into  the 
question  whether  a  particular  report  format  is  a  function  of  the  task  required  of  a  decision¬ 
maker.  To  investigate  Davis  invents  two  constructs:  (1)  efficiency  of  different  forms  of 
presentation  and  (2)  the  effectiveness  with  which  information  is  presented  to  the  decision¬ 
maker.  He  formulates  two  hypotheses  to  test  his  concept  of  format  and  task.  The  results 
of  the  statistical  analysis  of  the  experimental  responses  precludes  rejection  of  his 
hypotheses.  In  fact  he  identifies  an  interaction  between  the  two  which  implies  that  the 
relative  efficiency  and  effectiveness  of  differing  forms  of  presentation  are  affected  by  the 
question  (task)  with  which  the  decision-maker  is  confronted.  The  results  show  that  the 
task  (decision  problem  or  question)  and  the  form  it  is  presented  in  interact  to  affect  the 
decision-maker’s  performance—  the  time  and  accuracy  with  which  he  completes  the  task. 
Additionally,  Davis  concludes  that  no  one  particular  form  of  problem  presentation  is  best 


for  all  decision  situations. 


Critique 


This  critique  examines  Davis’  experimental  study  concerning  report  formats  and  the 
decision-maker’s  task.  I  will  attempt  to  summarize  his  key  concepts,  constructs,  results, 
and  his  findings.  Additionally,  I  will  point  out  the  areas  that  seem  to  overlook  items  of 
import.  Specific  criteria  for  evaluation  include  reasonable  steps  in  logic:  do  the  ideas  flow 
logically  and  are  they  logically  exhaustive,  meaning  all  logical  explanation  are  considered; 
does  his  experiment  adequately  weigh  the  threats  to  both  internal  and  external  validity? 

“The  objective  of  this  study  is  to  investigate  whether  the  information  which  a  decision 
maker  wishes  to  extract  from  an  information  presentation  is  a  task  eharacteristic  which 
affects  the  appropriateness  of  different  report  formats.”  (Davis,  1989:  496).  Davis 
defines  the  task  in  terms  of  the  questions  to  be  answered.  The  task  could  be  simple  data 
extraction  from  a  graphical  presentation:  Which  company  had  the  highest  gross  profits  in 
1995?  Or  the  task  might  be  a  complex  decision  problem:  Based  on  the  information 
provided  in  the  graphical  presentation,  should  we  continue  to  make  this  product  or  should 
we  contract  its  production.  Performance  is  then  measured  by  the  amount  of  time  taken  to 
answer  the  questions,  i.e.  the  time  to  complete  the  task. 

To  test  the  concept  that  performance  is  a  function  of  both  time  and  accuracy,  Davis 
adopts  a  construct  utilizing  both  the  efficiency  of  different  forms  of  presentation  to 
differing  tasks  and  the  effectiveness  of  communicating  information  to  a  decision-maker  via 
graphic  forms  versus  other  methods  (tabular,  report  text,  etcetera).  Two  hypotheses  were 
designed  to  test  these  constructs: 
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Hi:  The  form  of  presentation  which  allows  a  question  to  be 
answered  in  the  least  amount  of  time  will  be  different  for 
questions  of  different  levels  of  complexity. 


Hi:  The  form  of  presentation  which  results  in  the  most 
accurate  answers  to  a  question  will  be  different  for 
questions  of  different  levels  of  complexity.  (Davis,  1989: 

498). 

Davis  chose  a  laboratory  experiment  to  conduct  this  study,  testing  the  above  two 
hypotheses.  Thirty  MBA  students  participated,  one  at  a  time,  making  this  a  cross- 
sectional  sample. 

The  experiment  itself  is  “A  full-factorial,  within-subject  experimental  design...;  each 
subject  receives  all  twenty  experimental  treatments  (five  questions  manipulated  over  four 
forms  of  presentation)  in  a  different  random  order.”  (Davis,  1989:  498).  The  questions 
are  financial  in  nature,  accompanied  by  various  graphs  that  contain  the  question’s  answer. 
The  design  of  the  experiment  appears  to  be  valid.  Internal  validity  is  supported  in  that:  (1) 
the  instrument  does  seem  to  measure  what  it  purports  to  measure,  (2)  the  differences  in 
responses  (measures)  reflect  true  differences,  and  (3)  the  instrument  is  a  structured  one 
and  research  (Abdolmohammadi  &  Wright,  1987)  indicates  that  the  performance  of  the 
sample  students  are  representative  of  real  world  decision-makers,  utilizing  this  type  of 
financial  data  and  reaching  similar  answers. 

The  variables  consisted  of  one  control,  two  independent,  and  two  dependent;  the 
control  variable  was  the  information  set.  The  two  independent  variables  consisted  of  the 
form  of  the  presentation  and  its  corresponding  question.  The  dependent  variables  were 


the  time  the  decision-maker  needed  to  answer  a  question  and  the  accuracy  with  which  he 
did  so. 

The  responses  were  analyzed  with  multivariate  analysis  of  variance.  The  results  show 
that  “there  is  at  least  one  significant  effect.”  (Davis,  1989:  501).  The  model  utility  tests 
for  the  hypotheses 

Show  that  both  the  main  and  interactive  effects  of  the 
question  to  be  answered  and  the  form  of  presentation  are 
significant.  This  is  true  regardless  of  whether  performance 
is  measured  in  terms  of  the  time  required  to  answer  the 
question  or  the  accuracy  of  the  question-answers.  (Davis, 

1989;  501). 

The  results  support  the  contention  that  the  form  of  a  presentation  and  the  question  to  be 
answered  (the  task)  both  have  an  effect  on  the  performance  of  a  decision-maker.  Another 
important  result  was  seen  in  the  interactive  effects;  this  interaction  implies  that  the  relative 
efficiency  and  effectiveness  of  differing  forms  of  presentation  are  affected  by  the  question 
(task)  with  which  the  decision-maker  is  confronted.  Furthermore,  this  implies  that  no  one 
mode  of  presentation  is  better  in  all  decision  tasks.  Certain  graphics  may  better  aid 
decision-makers  with  one  type  of  task,  while  other  graphical  methods  will  benefit  a 
different  question’s  resolution.  In  Davis’  words,  “...the  most  appropriate  form  of 
presentation...  was  different  for  different  questions  and  support  both  hypothesis  1  and 
hypothesis  2.”  (Davis,  1989;  502). 

Davis’  conclusion  states  “The  results  indicate  that  the  most  appropriate  method  of 
presenting  financial  information  is  dependent  on  the  decision-maker’s  question;  different 
forms  of  presentation  are  most  appropriate  for  different  questions.”  (Davis,  1989;  504). 
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The  conclusion  that  different  forms  of  presentation  are  best  for  differing  tasks  seems  valid 
for  the  generic  financial  data  used  in  this  particular  experiment;  however,  the 
generalizability  of  such  a  study  is  limited  to  the  types  of  financial  presentations  and 
questions  it  represents. 

While  the  design  of  the  study  is  its  strength,  its  weakness  is  generalizability.  The 
strength  of  its  internal  validity  was  previously  discussed.  External  validity  is  apparent  in 
that  the  type  of  tasks  and  graphs  used  are  prevalent  in  business  and  industry.  However, 
the  conclusion  drawn  by  Davis,  as  indicated  by  the  study,  is  not  generalizable  across  all 
types  of  data  extraction  and  decision  problems.  This  weakness  is  important  and  goes 
directly  to  the  studies  external  validity.  Davis’  results  should  be  considered  tentative. 
Further  studies  should  attempt  to  broaden  the  scope  of  graphic  presentation  as  it  relates  to 
decision  tasks.  Until  that  time  no  general  statements  about  the  appropriateness  of 
different  presentation  formats  may  be  made. 
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Appendix  B.  Outsourcing  Presentations 
Control 


Outsourcing 

Decision 


In-house  Net*  Costs  vs.  Contraetor  Bids 

^Net  =  difference  between  revenue  and  costs. 

Costs 
S (000) 

140 

135 
130 
125 
120 
115 
110 

1990  1991  1992  1 993  1 994  1 995 

Tlie  contractor  has  underbid  our  costs  eveiy  year. 

Disagree  Agree 
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Hospital  Cafeteria 
Customer  Satisfaction 

l=poor  10=  excellent 


1990  1991  1992  1993  1994  1995 

Customer  satisfaction  witli  hospital  food  service  &  quality  declined  each  year  over  tlie  period  considered. 
Disagi'ee  Agree 


Contractor  Reported 
Customer  Satisfaction 

1=  poor  10=  excellent 


1990  1991  1992  1993  1994  1995 


Customer  satisfaction  with  tlie  contractor's  food  ser\nce  &  qiuility  has  consistently  declined. 


Percentage  of  Cafeteria  Employees  Retamed 
by  the  Contractor  and  the  Hospital  if  the 
Food  Service  Function  is  Contracted  Out 


Contractor 


Hospital 


No  employees  would  lose  tlieir  jobs  as  a  result  of  contracting  out  die  food  ser\dce  operation. 


Based  solely  on  the  information 
provided,  should  the  Food  Service 
operation  be  contracted  out? 


Without  changing  any  of  your 
previous  responses,  you  may 
review  the  six  graphs.  The 
graphs  will  be  shown  again  in  the 
same  sequence  as  before. 
Again,  please  do  not  change  any 
of  your  previous  responses. 


In-house  Net*  Costs  vs.  Contraetor  Bids 

*Net  =  difference  between  revenue  and  costs. 

Costs 
$  (000) 

140  n - ^ -  I - 


1990  1991  1992  1993  1994  1995 


Percentage  of  Cafeteria  Employees  Retained 
by  the  Contractor  and  the  Hospital  if  the 
Food  Service  Function  is  Contracted  Out 


Based  upon  your  review  of  the 
graphs,  would  you  make  the 
same  decision? 


No  Yes 
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Please  complete  the 
attached  survey  and 
return  the  entire 
package. 

Thank  you. 
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Experimental 


,se  Net  Costs  vs.  Cosiimotor  Bids 


S  Inhouse 
Net  Costs 

□  Contractor 
Bids 


Ute  coiitractoir  has  imdef^id  out  costs  everv  year. 
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Contractor  Hospital  Total 

No  employees  would  lose  Iheir  jobs  a  resuU  of  coiitraetiug  out  the  food  s^yice  operation. 
Disagree  '  "  Agree 


solely  on  the  information 
^  provided,  should  the  Food  Serviee 

operation  be  cdhifaeted  out? 

•  ,  Ng^  ■  ■  ? 


89 


1 


T 


if"? 


-f’  i  ^  ,,, 

;  A  ^sa'  ,:C®=  At  %=.'  A  A'  i.  ii^#* 


tt^l  »|3-M 


^■pi. 


:  jsi 


-S'l  lr-  It'S*.' 
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Appendix  C.  Benchmarking  Presentations 
Control 


Benchmarking 

Decision 


Distribution  Cost  Per  Item 


$1.50 

$1.45 

$1.40 

$1.35 

$1.30 

$1.25 

$1.20 


□  Ours 


*94Qtr3  '94Qtr4  '95  Qtr  1  '95Qtr2  '95Qtr3  '95  Qtr  4 


Our  distribution  cost  per  item  has  remained  undianged. 


Disagree 


Agree 
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Distribution/Delivery  Time 
Customer  Satisfaction 

1=  poor  10=  excellent 


Customer  stitisfciction  witli  distribution/deliveiy^  time  luis  declined  over  the  period  being  considered. 


Disagree 


Agree 


Based  solely  on  the  information 
provided,  should  the  Distribution 
Process  be  evaluated  against  the 
world’s  benchmark  distributor? 

No  Yes 
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Without  changing  any  of  your 
previous  responses,  you  may 
review  the  six  graphs.  The 
graphs  will  be  shown  again  in  the 
same  sequence  as  before. 
Again,  please  do  not  change  any 
of  your  previous  responses. 


Distribution  Cost  Per  Item 

$1.50 
$1.45 
$1.40 
$1.35 
$1.30 
$1.25 
$1.20 

■94  Qtr  3  '94  Qtr  4  '95  Qtr  1  '95  Qtr  2  '95  Qtr  3  '95  Qtr  4 


□  Ours 
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Distribution/Delivery  Time 
Customer  Satisfaction 

1=  poor  10=  excellent 


’94Qtr3  •94Qtr4  '95  Qtr  1  '95  Qtr  2  '95  Qtr  3  '95  Qtr  4 


Based  upon  your  review  of  the 
graphs,  would  you  make  the 
same  decision? 


Please  complete  the 
attached  survey  and 
return  the  entire 
package. 

Thank  you. 
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Experimental 


Our  distribution  cost  per  item  has  remained  unchanged. 


Disagree 


Agree 


Oiir  stock  prices  have  declined  overtlie  period  under  consideration. 
Disagree  Agree 


*94  Qtr  3  *94  Qtr  4  *95  Qtr  1  *95  Qtr  2  *95  Qtr  3  *95  Qtr  4 

Our  share  of  tire  market  has  been  declining  over  the  past  tliree  quarters. 


‘94  Qtr  3  ‘94  Qtr  4  *95  Qtr  1  ‘96  Qtr  2  *95  Qir  3  ‘95  Qtr  4 


Customer  satisfaction  witli  distribution/delivery  time  has  declined  over  the  period  being  considei'ed. 
Disagree  Agree 


d’*' i.'  i-  I  '  p  '■ 

r  J: i  I  I  i  irt  ^  V  C  ) I  H  j r’f  ■'  ^  ?  H  J  M  ^  I  i C  i  ^  ~ 

\  1  ^  jf^l  ?  I  I  '  f  •  "^1  I  V""  ^  ■  f  f ‘  if  T% 

^  ^s‘  U  'x-hJ''  --VAi  ^  V:  %  'Wr  &  'xA  x  ir!  I  rj  ii  vi  %.,,M  ‘i,,.  \[  ,s  t 

j  ;f=>.  «  ,ii<T^,-  ^  J  ■>'''''1  5‘''''^'^S  #'•"«;  TX- 4  'k'  ^'"'1 

...  .  .  ' 

'l/i/f'");'  f'  1  f ■'  CJ  jf '1  f ‘I  f"';  I'"!'  I'T"!  'i|  I C;  I"  f“'f  f t  j  T  f  j  f '  / 


No  Yes 
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Plij 


□  Closest 
Competitor 


Ill 


Appendix  D,  Outsourcing  Package 


Outsourcing 

Decision 
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Decision  Problem 


You  are  the  Chief  Executive  Officer  (CEO)  of  a  major  metropolitan  hospital.  The  board  of 
regents  has  charged  you  with  the  task  of  determining  whether  the  hospital  should  continue  to 
operate  its  cafeteria.  Based  solely  on  the  information  contained  in  the  following  graphs,  you 
must  decide  whether  to  maintain  in-house  operation  of  the  cafeteria  or  contract  out  its 
function  to  a  commercial  food  service  contractor. 

Currently,  the  cafeteria  is  run  completely  by  the  hospital  in  the  hospital’s  facilities.  If  the 
cafeteria  function  were  contracted  out,  all  responsibilities  for  its  operation  would  be  assumed 
by  the  contractor.  However,  the  location  of  the  cafeteria  would  remain  unchanged. 

The  first  three  graphs  presented  depict  financial  trends  impacting  your  decision,  while  the 
final  three  show  qualitative  trends.  Assume  that  these  trends  will  continue  if  the  hospital 
makes  no  changes.  Your  decision  should  be  based  solely  on  the  information  contained  in  the 
following  six  graphs.  Do  you  contract  out  the  hospital’s  food  service  function? 


Decision  Rules 

Assuming  that  the  six  graphs  that  follow  contain  ^  the  information  necessary  to  make  an 
informed  decision,  circle  the  printed  response  that  corresponds  to  your  answer  for  each  of  the 
graphs.  Your  response  tasks  are  two  fold:  (1)  For  each  slide,  you  will  be  asked  to  indicate 
disagreement  or  agreement  with  the  slide’s  corresponding  statement.  Your  response  should 
indicate  your  disagreement  or  agreement  with  that  particular  graph.  (2)  At  the  end  of  the 
presentation,  you  will  be  asked  to  make  an  overall  decision  based  on  all  the  information 
presented.  This  decision  should  be  based  solely  upon  the  six  graphs.  No  information  external 
to  that  presented  in  the  graphs  should  be  considered  in  any  of  your  decision-making. 

This  is  not  a  test  and  your  name  will  not  be  recorded.  However,  this  is  a  timed 
experiment.  You  will  be  given  20  seconds  per  graph  to  view  each  one  and  respond.  The 
computer  will  automatically  advance  the  slides  you  will  be  using  to  make  your  decisions. 

There  will  be  a  statement  at  the  bottom  of  each  slide.  The  statement  will  also  be  on  the 
answer  sheet.  Please  circle  the  response  (disagree  or  agree)  that  corresponds  to  your 
response  for  each  of  the  six  slides  and  for  your  overall,  final  decision. 

The  total  length  of  this  experiment  should  not  exceed  10  minutes. 


Stop! 

Please,  do  not  continue  until  instructed  to  do  so. 
Thank  you  in  advance  for  your  conscientious  participation. 


Answer  Sheet 


(Circle  yow  responses)  In-house  Net  Costs  vs.  Contractor  Bids 

Disagree  Agree  1)  The  contractor  has  underbid  our  costs  every  year. 


Total  Revenue  vs.  Total  Costs 

Disagree  Agree  2)  Total  costs  have  exceeded  total  revenue  for  each  of  the  past 

six  years. 

Profit  and  Loss  (P&L) 

Disagree  Agree  3)  The  hospital  food  service  has  sustained  increased  losses  over 

the  past  six  years. 


Hospital  Cafeteria  Customer  Satisfaction 

Disagree  Agree  4)  Customer  satisfaction  with  hospital  food  service  &  quality 

declined  each  year  over  the  period  considered. 


Contractor  Reported  Customer  Satisfaction 

Disagree  Agree  5)  Customer  satisfaction  with  the  contractor’s  food  service  & 

quality  has  consistently  declined. 


Percentage  of  Cafeteria  Employees  Retained 
by  the  Contractor  and  the  Hospital  if  the  Food  Service  Function  is  Contracted  Out 

Disagree  Agree  6)  No  employees  would  lose  their  jobs  as  a  result  of 

contracting  out  the  food  service  operation. 


Final  Decision 


Based  solely  on  the  information  provided, 
should  the  food  service  operation  be  contracted  out? 


Without  changing  any  of  your  previous  responses,  you  may  review  the  six  graphs.  The 
graphs  will  be  shown  again  in  the  same  sequence  as  before. 

Again,  please  do  n^  change  any  of  your  previous  responses. 


Reviewed  Final  Decision 


Based  upon  your  review  of  the  graphs, 
would  you  make  the  same  decision? 

No  Yes 


Please  complete  the  following  informational  questionnaire  and  return  the  entire  package. 


Questionnaire 


This  questionnaire  contains  statements  and  questions  that  concern: 

a)  your  background  information  for  demographics, 

b)  your  level  of  experience  with  graphs,  and 

c)  the  experiment  just  completed. 


Please  circle  the  most  appropriate  answer  from  the  choices  provided. 

1.  What  is  your  gender?  Female  Male 


2.  What  is  your  Education  Level? 


High  School 
Some  College 
Associates  Degree 


Baccalaureate  Degree  Doctoral  Degree 

Some  Graduate  Courses 
Masters  Degree 


3.  Are  you  color  blind?  No  Yes 


4.  Have  you  ever  had  any  training  with  graph  construction  and/or  interpretation? 
{Circle  all  that  apply). 

a)  Yes,  formal  training  on  graph  construction. 

b)  Yes,  formal  training  on  graph  interpretation. 

c)  Yes,  informal  training  on  graph  construction. 

d)  Yes,  informal  training  on  graph  interpretation. 

e)  No  formal  or  informal  training  on  graph  construction  or  interpretation. 


5.  How  often  do  you  use  graphs  in  decision-making? 

Every  day  Once  a  month  Never 

Every  other  day  Once  a  quarter 
Once  a  week  Once  a  year 

6.  How  often  do  you  see  pictures  on  graphs  (i.e.  magazines,  newspapers,  briefings,  etc.)? 

Every  day  Once  a  month  Never 

Every  other  day  Once  a  quarter 

Once  a  week  Once  a  year 

7.  Were  the  instructions  clear  and  easy  to  follow?  Comments; _ 

No  Yes 

8.  What  was  your  level  of  interest  in  the  experimental  decision? 

1  2  3  4  5  6  7 

not  interested  very  interested 

9.  The  individual  experimental  questions  (those  on  each  graph)  were  easy  to  understand. 

1  2  3  4  5  6  7 

disagree  agree 

10.  Based  solely  on  the  information  provided,  how  confident  are  you  that  you  made  the 
appropriate  overall,  final  decision? 

1  2  3  4  5  6  7 

not  confident  confident 

1 1 .  Did  you  have  any  previous  knowledge  of  this  experiment? 

No  Yes 


Thank  you  for  participating  in  this  experiment! 


Benchmarking 

Decision 
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Decision  Problem 


You  are  the  Chief  Executive  Officer  (CEO)  of  a  large  industrial  firm  that  produces  and 
distributes  ball  bearings.  Over  the  past  six  quarters  (1  V2  years)  it  seems  that  something  is 
wrong  with  the  firm’s  distribution  process.  No  one  within  the  company  can  exactly  identify 
the  problem.  Based  solely  on  the  information  contained  in  the  following  graphs,  you  must 
decide  whether  to  continue  working  the  issue  in-house  or  consult  the  world’s  most  efficient 
distributor  for  assistance  in  evaluating/benchmarking  your  company’s  distribution  process. 
Although  this  benchmark  company  doesn’t  deal  in  ball  bearings,  it  should  be  able  to  identify 
and  help  correct  the  problem. 

Currently,  your  distribution  difficulties  are  suspected  to  be  impacting  the  company’s 
performance  and  competitiveness.  Your  closest  competitor  in  the  ball  bearing  business  is 
gaining  ground  on  your  firm.  Five  of  the  following  six  graphs  portray  quantitative  trends 
concerning  the  company’s  performance;  three  of  the  five  depict  the  firm’s  competitiveness  as 
compared  with  your  company’s  closest  competition.  The  final  graph  relates  a  qualitative 
trend.  Assume  that  these  trends  will  continue  if  the  firm  requests  no  assistance. 

Although  the  benchmark  in  distribution  operates  in  an  entirely  different  business,  it  is 
reasonable  to  assume  that  this  company  can  help  identify  your  company’s  distribution 
troubles.  This  benchmark  company,  if  consulted,  is  willing  to  help  you  evaluate  your 
processes  and  aid  in  correcting  your  problems.  The  decision  you’re  faced  with  is:  do  you  ask 
this  company  to  benchmark  your  process? 


Decision  Rules 

Assuming  that  the  six  graphs  that  follow  contain  ah  the  information  necessary  to  make  an 
informed  decision,  circle  the  printed  response  that  corresponds  to  your  answer  for  each  of  the 
graphs.  Your  response  tasks  are  two  fold;  (1)  For  each  slide,  you  will  be  asked  to  indicate 
disagreement  or  agreement  with  the  slide’s  corresponding  statement.  Your  response  should 
indicate  your  disagreement  or  agreement  with  that  particular  graph.  (2)  At  the  end  of  the 
presentation,  you  will  be  asked  to  make  an  overall  decision  based  on  all  the  information 
presented.  This  decision  should  be  based  solely  upon  the  six  graphs.  No  information  external 
to  that  presented  in  the  graphs  should  be  considered  in  any  of  your  decision-making. 

This  is  not  a  test  and  your  name  will  not  be  recorded.  However,  this  is  a  timed 
experiment.  You  will  be  given  20  seconds  per  graph  to  view  each  one  and  respond.  The 
computer  will  automatically  advance  the  slides  you  will  be  using  to  make  your  decisions. 

There  will  be  a  statement  at  the  bottom  of  each  slide.  The  statement  will  also  be  on  the 
answer  sheet.  Please  circle  the  response  (disagree  or  agree)  that  corresponds  to  your 
response  for  each  of  the  six  slides  and  for  your  overall,  final  decision. 

The  total  length  of  this  experiment  should  not  exceed  10  minutes. 

Stop! 

Please,  do  not  continue  until  instructed  to  do  so. 

Thank  you  in  advance  for  your  conscientious  participation. 


Answer  Sheet 


(Circle  your  responses)  Distribution  Cost  Per  Item 

Disagree  Agree  1)  Our  distribution  cost  per  item  has  remained  unchanged. 

Stock  Price  Per  Share 

Disagree  Agree  2)  Our  stock  prices  have  declined  over  the  period  under 

consideration. 

Market  Share 

3)  Our  share  of  the  market  has  been  declining  over  the  past 
three  quarters. 

Cycle  Time 

4)  Cycle  time  has  decreased  since  ‘95  Qtr  1. 

Inventory  Turnover  Time 

5)  The  time  needed  to  go  through  100%  of  stocked  inventory 
is  declining. 

Distribution/Deliverv  Time  Customer  Satisfaction 

6)  Customer  satisfaction  with  distribution/delivery  has  declined 
over  the  period  being  considered. 

Final  Decision 

Based  solely  on  the  information  provided, 

should  the  distribution  process  be  evaluated  against  the  world’s  benchmark  distributor? 


Disagree  Agree 


Disagree  Agree 


Disagree  Agree 


Disagree  Agree 


Without  changing  any  of  your  previous  responses,  you  may  review  the  six  graphs.  The 
graphs  will  be  shown  again  in  the  same  sequence  as  before. 

Again,  please  do  not  change  any  of  your  previous  responses. 


Reviewed  Final  Decision 

Based  upon  your  review  of  the  graphs, 
would  you  make  the  same  decision? 

No  Yes 


Please  complete  the  attached  informational  questionnaire  and  return  the  entire  package. 


Questionnaire 


This  questionnaire  contains  statements  and  questions  that  concern: 

a)  your  background  information  for  demographics, 

b)  your  level  of  experience  with  graphs,  and 

c)  the  experiment  just  completed. 

Please  circle  the  most  appropriate  answer  from  the  choices  provided. 

1 .  What  is  your  gender?  Female  Male 


2.  What  is  your  Education  Level? 


High  School 
Some  College 
Associates  Degree 


Baccalaureate  Degree  Doctoral  Degree 

Some  Graduate  Courses 
Masters  Degree 


3.  Are  you  color  blind?  No  Yes 


4.  Have  you  ever  had  any  training  with  graph  construction  and/or  interpretation? 
{Circle  all  that  apply). 

a)  Yes,  formal  training  on  graph  construction. 

b)  Yes,  formal  training  on  graph  interpretation. 

c)  Yes,  informal  training  on  graph  construction. 

d)  Yes,  informal  training  on  graph  interpretation. 

e)  No  formal  or  informal  training  on  graph  construction  or  interpretation. 


5.  How  often  do  you  use  graphs  in  decision-making? 

Every  day  Once  a  month  Never 

Every  other  day  Once  a  quarter 
Once  a  week  Once  a  year 

6.  How  often  do  you  see  pictures  on  graphs  (i.e.  magazines,  newspapers,  briefings,  etc.)? 

Every  day  Once  a  month  Never 

Every  other  day  Once  a  quarter 

Once  a  week  Once  a  year 

7.  Were  the  instructions  clear  and  easy  to  follow?  Comments: _ 

No  Yes 

8.  What  was  your  level  of  interest  in  the  experimental  decision? 

1  2  3  4  5  6  7 

not  interested  very  interested 

9.  The  individual  experimental  questions  (those  on  each  graph)  were  easy  to  understand. 

1  2  3  4  5  6  7 

disagree  agree 

10.  Based  solely  on  the  information  provided,  how  confident  are  you  that  you  made  the 
appropriate  overall,  final  decision? 

1  2  3  4  5  6  7 

not  confident  confident 

1 1 .  Did  you  have  any  previous  knowledge  of  this  experiment? 

No  Yes 


Thank  you  for  participating  in  this  experiment! 


Outsourcing  Decision  Graphs _ Benchmarking  Decision  Graphs 


Graph  1  =  In-house  Net  Cost  vs.  Contractor  Bids  Distribution  Cost  Per  Item 

Graph  2  =  Total  Revenue  vs.  Total  Costs  Stock  Price  Per  Share 

Graph  3  =  Profit  and  Loss  (P&L)  Market  Share 

Graph  4  =  Hospital  Cafeteria  Customer  Satisfaction  Cycle  Time 
Graph  5  =  Contractor  Reported  Customer  Satisfaction  Inventory  Turnover  Time 
Graph  6  =  Percentage  of  Cafeteria  Employees  Retained  Distribution/Delivery  Customer 
by  the  Contractor  and  the  Hospital  if  the  Satisfaction 

Food  Service  is  Contracted  Out 


Appendix  G.  Response  Data 


The  four  datasheets  that  make  up  this  appendix  contain  the  responses  obtained 
from  this  study’s  experiments.  “S1-S6”  represent  responses  to  the  statements  on  the  six 
graphs  used  in  the  decision  process.  “Q1-Q12”  correspond  to  the  post  test  questionnaire 
responses.  “D”,  “R”,  and  “C”  are  the  decision  reached  by  the  subject,  the  reviewed 
decision,  and  an  indication  of  consistency  between  the  previous  two,  respectively. 
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Experimental  Group  Charts:  Benchmarking 


Demographics 
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Appendix  H,  Format  for  Importing  Outsourcing  Data  Into  Statgraphics 

The  datasheet  that  makes  up  this  appendix  contains  the  responses  obtained  from 
this  study’s  experiments.  This  data  is  a  transposition,  in  part,  of  that  in  Appendix  G.  This 
format  was  necessary  to  facilitate  the  use  of  Statgraphics.  The  outsourcing  decision 
problem  had  60  subjects. 

“DP”  represents  the  decision  problem  and  was  categorized  as  “0”  for  outsourcing 
and  “1”  for  benchmarking;  compare  Appendices  H  and  I.  “GP”  represents  the  group  with 
“C”  indicating  the  control  and  “E”  the  experimental  group.  “S1-S6”  represent  the 
responses  to  the  statements  on  the  six  graphs  used  in  the  decision  process.  “Q1-Q12” 
correspond  to  the  post  test  questionnaire  responses.  “D,”  “R,”  and  “C”  are  the  decision 
reached  by  the  subject,  the  reviewed  decision,  and  an  indication  of  consistency  between 
the  previous  two,  respectively. 
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Appendix  I.  Format  for  Importing  Benchmarking  Data  Into  Statgraphics 


# 


The  datasheet  that  makes  up  this  appendix  contains  the  responses  obtained  from 
this  study’s  experiments.  This  data  is  a  transposition,  in  part,  of  that  in  Appendix  G.  This 
format  was  necessary  to  facilitate  the  use  of  Statgraphics.  The  benchmarking  decision 
problem  had  87  subjects. 

“DP”  represents  the  decision  problem  and  was  categorized  as  “0”  for  outsourcing 
and  “1”  for  benchmarking.  Compare  Appendices  H  and  I.  “GP”  represents  the  group 
with  “C”  indicating  the  control  and  “E”  the  experimental  group.  “S1-S6”  represent  the 
responses  to  the  statements  on  the  six  graphs  used  in  the  decision  process.  “Q1-Q12” 
correspond  to  the  post  test  questionnaire  responses.  “D”,  “R”,  and  “C”  are  the  decision 
reached  by  the  subject,  the  reviewed  decision,  and  an  indication  of  consistency  between 
the  previous  two,  respectively. 


Obs 

DP 

B1 

S2 

S3 

S4 

S5 

S6 

□ 

m 

Q2 

Q3 

B 

Q7 

m 

Qll 

1 

1 

D 

0 

0 

0 

0 

0 

□ 

B 

□ 

3 

0 

B 

0 

D 

a 

0 

2 

1 

D 

0 

1 

0 

1 

B 

D 

D 

D 

2 

0 

B 

B 

B 

0 

B 

B 

D 

0 

B 

1 

C 

0 

D 

0 

1 

0 

□ 

B 

1 

1 

0 

B 

B 

2 

B 

B 

4 

D 

0 

B 

C 

1 

□ 

0 

0 

0 

B 

0 

0 

4 

0 

B 

B 

0 

B 

B 

3 

D 

0 

B 

c 

1 

0 

0 

1 

D 

D 

0 

0 

3 

0 

B 

0 

3 

4 

0 

6 

1 

c 

0 

0 

0 

0 

B 

B 

0 

4 

0 

0 

0 

B 

0 

5 

5 

0 

7 

1 

c 

□ 

D 

D 

B 

B 

B 

B 

B 

1 

1 

0 

0 

B 

0 

6 

0 

8 

1 

□ 

0 

B 

0 

B 

B 

n 

B 

0 

3 

0 

7 

6 

7 

0 

9 

1 

□ 

0 

B 

0 

B 

B 

D 

B 

10 

3 

4 

0 

7 

6 

7 

0 

10 

1 

0 

1 

B 

B 

1 

1 

B 

0 

1 

4 

4 

0 

11 

1 

0 

0 

B 

B 

0 

0 

B 

0 

1 

2 

6 

0 

7 

0 

B 

1 

B 

D 

0 

1 

B 

0 

B 

B 

1 

4 

0 

1 

B 

0 

B 

1 

B 

□ 

0 

1 

B 

1 

D 

D 

0 

7 

4 

B 

0 

B 

B 

□ 

0 

0 

B 

1 

B 

B 

1 

0 

3 

B 

0 

B 

n 

C 

n 

□ 

□ 

B 

B 

B 

D 

3 

0 

B 

B 

3 

0 

2 

o 

B 

B 

c 

D 

□ 

1 

B 

B 

B 

B 

3 

0 

B 

B 

3 

0 

4 

7 

o 

B 

D 

c 

0 

1 

D 

0 

0 

B 

B 

B 

0 

2 

0 

0 

0 

3 

0 

B 

B 

0 

m 

B 

c 

0 

'l 

1 

0 

0 

B 

B 

B 

0 

6 

3 

4 

0 

B 

B 

0 

B 

c 

□ 

D 

□ 

B 

B 

B 

B 

B 

6 

0 

3 

1 

3 

0 

6 

6 

6 

0 

B 

c 

0 

B 

B 

0 

B 

B 

2 

0 

B 

3 

3 

0 

7 

B 

7 

0 

B 

B 

0 

B 

B 

0 

0 

B 

1 

0 

B 

0 

2 

0 

7 

B 

7 

0 

22 

B 

0 

B 

0 

0 

0 

B 

B 

D 

2 

B 

3 

4 

0 

5 

B 

7 

0 

23 

B 

c 

0 

B 

B 

B 

0 

B 

a 

2 

B 

0 

2 

0 

6 

B 

7 

0 

c 

D 

D 

0 

□ 

B 

B 

B 

D 

B 

B 

0 

B 

7 

o 

B 

c 

D 

□ 

B 

0 

0 

B 

B 

B 

Q 

B 

B 

B 

B 

B 

7 

o 

c 

B 

D 

B 

0 

0 

B 

B 

B 

B 

0 

B 

B 

B 

6 

o 

132 


lO 

IjjlMljH 

0 

3 

0 

5 

o 

o 

a 

lo 

a 

o 

a 

o 

D 

o 

D 

0 

D 

o 

a 

o 

1 

0 

2 

0 

o 

D 

o 

D 

o 

4 

o 

1 

o 

D 

o 

D 

D 

o 

4 

0 

2 

0 

Obs  DP  GP  SI  S2  S3  S4  S5 
66  1  E  1  1  0  1  0 


0 


Q3|Q4|Q5  Q6  Q7  Q8  Q9  QIO  Qll 
0  0  0  0  0  6  6  15  14 


_4 _ 0 _ 6 _ ^ 

5  I  0  5  3 

0  4  4 


0 _ 3 _ 4 _ 3 _ 0 _ 4 _ 4 

0  0  0  4  0  6  7 


DODD 

0 

□DD 

0 

6 

0 

□ODD 

0 

□DQ 

1 

4 

0 

0 

1 

oe 

0 

0 

□□nn 

2 

OD 

0 

no 

0 

0 

□DBD 

2 

DD 

3  0 


4 _ 0__^_6 _ 6 _ 0 _ 4 _ 7 _ 7 

20203143  4 


1 

ma 

E 

0 

0 

0 

0 

0 

DEDD 

6 

0 

0 

4 

DDD 

5 

5 

3 

0 

I 

an 

E 

0 

0 

0 

0 

0 

EDBD 

2 

0 

0 

0 

DD 

6 

5 

6 

3 

0 

1 

a 

1 

E 

0 

0 

0 

0 

0 

0 

□Bl 

0 

6 

2 

0 

6 

0 

6 

B 

7 

0 

1 

a 

1 

E 

0 

0 

0 

0 

0 

0 

□□I 

0 

4 

3 

5 

0 

DD 

B 

3 

0 

1 

E 

0 

0 

0 

0 

B 

B 

D 

4 

0 

0 

0 

6 

0 

1 

7 

7 

1 

0 

1 

E 

0 

0 

0 

0 

B 

B 

D 

4 

0 

0 

1 

6 

0 

7 

7 

6 

1 

0 

34 


Appendix  J.  Statgraphics  Output 


Outsoiirchig  Decision  Problem 


Kmskal-Wallis  analysis  of  Confidence  by  Group 

Level 

Sample  Size 

Average  Rank 

C 

29 

29.6034 

E 

30 

30.3833 

Test  statistic 

=  0.0322734 

Significance  level  =  0.857429 

Kruskal-Wallis  analysis  of  Decision  by  Group 

Level  Sample  Size 

Average  Rank 

C  29 

27.4138 

E  28 

30.6429 

Test  statistic  =  1.0205  Significance  level  =0.312401 


Kruskal-Wallis  analysis  of  Decision  by  Gender 

Level 

Sample  Size 

Average  Rank 

0 

34 

28.3676 

1 

23 

29.9348 

Test  statistic 

=  0.231485 

Significance  level  =  0.630425 

Kruskal-Wallis  analysis  of  Decision  by  Training 


Level 

Sample  Size 

Average  Rank 

0 

16 

26.0625 

1 

7 

30.6429 

2 

7 

26.5714 

3 

11 

30.2727 

4 

5 

33.9000 

7 

10 

31.0500 

10 

1 

22.5000 

Test  statistic 

=  2.88818 

Significance  level  =  0.822745 

Kruskal-Wallis  analysis  of  Confidence  by  Gender 


Level 

Sample  Size 

Average  Rank 

0 

35 

32.3714 

1 

24 

26.5417 

Test  statistic  =  1.741 19  Significance  level  =  0.186988 


Kruskal-Wallis  analysis  of  Confidence  by  Training 


Level 

Sample  Size 

Average  Rank 

0 

17 

32.4706 

1 

8 

27.0625 

2 

7 

24.7143 

3 

11 

25.9091 

4 

5 

30.7000 

7 

10 

37.0000 

10 

1 

20.0000 

Test  statistic  =  4.1 1991  Significance  level  =  0.660453 


Kruskal-Wallis  analysis  of  Outsourcing  Graph  1  by  Group 

Level  Sample  Size 

Average  Rank 

C  29 

27.5000 

E  29 

31.5000 

Test  statistic  =  1.2  Significance  level  =0.273322 


Kruskal-Wallis  analysis  of  Outsourcing  Graph  2  by  Group 

Level  Sample  Size 

Average  Rank 

C  27 

26.5000 

E  29 

30.3621 

Test  statistic  =  3.93899  Significance  level  =  0.047179 


Kruskal-Wallis  analysis  of  Outsourcing  Graph  3  by  Group 


Level  Sample  Size  Average  Rank 


C  29  27.5000 

E  28  30.5536 


Test  statistic  =  3.22222  Significance  level  =  0.0726449 


Kruskal-Wallis  analysis  of  Outsourcing  Graph  4  by  Group 

Level  Sample  Size 

Average  Rank 

C  29 

27.0000 

E  29 

32.0000 

Test  statistic  =  1.83398  Significance  level  =  0.175659 
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Kruskal-Wallis  analysis  of  Outsourcing  Graph  5  by  Group 

Level  Sample  Size 

Average  Rank 

C  29 

28.5000 

E  29 

30.5000 

Test  statistic  =  1.05556  Significance  level  =  0.304231 


Kruskal-Wallis  analysis  of  Outsourcing  Graph  6  by  Group 

Level  Sample  Size 

Average  Rank 

C  29 

26.2586 

E  28 

31.8393 

Test  statistic  =  2.20141  Significance  level  =  0. 137885 


Kruskal-Wallis  analysis  of  Reviewed  Decision  by  Group 

Level  Sample  Size 

Average  Rank 

C  29 

29.4138 

E  28 

28.5714 

Test  statistic  =  0.091954  Significance  level  =0.761708 


Kruskal-Wallis  analysis  of  Reviewed  Decision  by  Gender 

Level  Sample  Size 

Average  Rank 

0  34 

27.8529 

1  23 

30.6957 

Test  statistic  =  1.00853  Significance  level  =  0.315256 
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Kruskal-Wallis  analysis  of  Reviewed  Decision  by  Training 


Level 

Sample  Size 

Average  Rank 

0 

16 

28.0625 

1 

7 

32.6429 

2 

7 

24.5000 

o 

11 

29.6818 

4 

5 

30.2000 

7 

10 

30.2000 

10 

1 

24.5000 

Test  statistic 

=  2.68965 

Significance  level  =  0.846669 

Benchmarking  Decision  Problem 


Kruskal-Wallis  analysis  of  Confidence  by  Group 

Level 

Sample  Size 

Average  Rank 

C 

40 

45.8250 

E 

46 

41.4783 

Test  statistic 

=  0.680786 

Significance  level  =  0.409316 

Kruskal-Wallis  analysis  of  Decision  by  Group 

Level 

Sample  Size 

Average  Rank 

C 

39 

43.3846 

E 

45 

41.7333 

Test  statistic 

=  0.333576  Significance  level  =0.563561 

Kruskal-Wallis  analysis  of  Decision  by  Gender 

Level 

Sample  Size 

Average  Rank 

46 

42.5652 

1 

38 

42.4211 

Test  statistic 

=  2.53242E-3 

Significance  level  =  0.959865 

Kruskal- Wallis  analysis  of  Decision  by  Training 


Level 

Sample  Size 

Average  Rank 

0 

45 

44.5333 

1 

5 

38.0000 

2 

6 

38.0000 

3 

11 

41.8182 

4 

2 

38.0000 

5 

1 

38.0000 

6 

1 

38.0000 

7 

8 

38.0000 

10 

5 

46.4000 

Test  statistic 

=  4.29178  Significance  level  =  0.829885 

Kruskal-Wallis  analysis  of  Confidence  by  Gender 

Level 

Sample  Size 

Average  Rank 

0 

48 

47.0938 

1 

38 

38.9605 

Test  statistic 

=  2.36274  Si 

gnificance  level  =0.124264 
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Kmskal-Wallis  analysis  of  Confidence  by  Training 


Level 

Sample  Size  Average  Rank 

0 

47 

39.1915 

1 

5 

51.6000 

2 

6 

41.2500 

3 

11 

46.6364 

4 

2 

11.7500 

5 

1 

17.0000 

6 

1 

76.0000 

7 

8 

58.0000 

10 

5 

60.0000 

Test  statistic 

=  13.737 

Significance  level  =0.0888819 

Kruskal-Wallis  analysis  of  Benchmarking  Graph  1  by  Group 

Level  Sample  Size 

Average  Rank 

C  41 

E  46 

43.3659 

44.5652 

Test  statistic  =  0. 120646 

Significance  level  =  0.728335 

Kruskal-Wallis  analysis  of  Benchmarking  Graph  2  by  Group 

Level  Sample  Size 

Average  Rank 

C  41 

E  46 

43.8049 

44.1739 

Test  statistic  =  0.0139634  Significance  level  =0.905936 


Kruskal-Wallis  analysis  of  Benchmarking  Graph  3  by  Group 


Level  Sample  Size  Average  Rank 


C  41  49.4756 

E  46  39.1196 


Test  statistic  =  6.24513  Significance  level  =  0.0124535 


Kruskal-Wallis  analysis  of  Benchmarking  Graph  4  by  Group 


Level  Sample  Size  Average  Rank 


C  40  41.2250 

E  46  45.4783 


Test  statistic  =  1.85477  Significance  level  =0.17323 


Kruskal-Wallis  analysis  of  Benchmarking  Graph  5  by  Group 


Level  Sample  Size  Average  Rank 


C  41  44.8049 

E  46  43.2826 


Test  statistic  =  0.282949  Significance  level  =  0.594775 


Kruskal-Wallis  analysis  of  Benchmarking  Graph  6  by  Group 

Level  Sample  Size 

Average  Rank 

C  41 

46.8049 

E  46 

41.5000 

Test  statistic  =  5.8834  Significance  level  =0.0152843 
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Kruskal-Wallis  analysis  of  Reviewed  Decision  by  Group 

Level 

Sample  Size 

Average  Rank 

C 

39 

42.7308 

E 

45 

42.3000 

Test  statistic 

=  0.0327416 

Significance  level  =0.85641 

Kruskal-Wallis  analysis  of  Reviewed  Decision  by  Gender 

Level 

Sample  Size 

Average  Rank 

46 

42.2391 

1 

38 

42.8158 

Test  statistic 

=  0.0584404 

Significance  level  =0.808978 

Kruskal-Wallis  analysis  of  Reviewed  Decision  by  Training 

Level 

Sample  Size 

Average  Rank 

0 

45 

43.2333 

1 

5 

39.5000 

2 

6 

39.5000 

J 

11 

43.3182 

4 

2 

39.5000 

5 

1 

39.5000 

6 

1 

39.5000 

7 

8 

39.5000 

10 

5 

47.9000 

Test  statistic 

=  3.24605  Significance  level  =0.917981 
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